Methods of Applied Mathematics For Engineers and Scientists [PDF]

Zitiervorschau

METHODS OF APPLIED MATHEMATICS FOR ENGINEERS AND SCIENTISTS Based on course notes from more than twenty years of teaching engineering and physical sciences at Michigan Technological University, Tomas Co’s engineering mathematics textbook is rich with examples, applications, and exercises. Professor Co uses analytical approaches to solve smaller problems to provide mathematical insight and understanding, and numerical methods for large and complex problems. The book emphasizes applying matrices with strong attention to matrix structure and computational issues such as sparsity and efficiency. Chapters on vector calculus and integral theorems are used to build coordinate-free physical models, with special emphasis on orthogonal coordinates. Chapters on ordinary differential equations and partial differential equations cover both analytical and numerical approaches. Topics on analytical solutions include similarity transform methods, direct formulas for series solutions, bifurcation analysis, LagrangeCharpit formulas, shocks/rarefaction, and others. Topics on numerical methods include stability analysis, differential algebraic equations, high-order finite-difference formulas, Delaunay meshes, and others. MATLAB implementations of the methods and concepts are fully integrated. Tomas Co is an associate professor of chemical engineering at Michigan Technological University. After completing his PhD in chemical engineering at the University of Massachusetts at Amherst, he was a postdoctoral researcher at Lehigh University, a visiting researcher at Honeywell Corp., and a visiting professor at Korea University. He has been teaching applied mathematics to graduate and advanced undergraduate students at Michigan Tech for more than twenty years. His research areas include advanced process control, including plantwide control, nonlinear control, and fuzzy logic. His journal publications span broad areas in such journals as IEEE Transactions in Automatic Control, Automatica, AIChE Journal, Computers in Chemical Engineering, and Chemical Engineering Progress. He has been nominated twice for the Distinguished Teaching Awards at Michigan Tech and is a member of the Michigan Technological University Academy of Teaching Excellence.

Methods of Applied Mathematics for Engineers and Scientists Tomas B. Co Michigan Technological University

cambridge university press Cambridge, New York, Melbourne, Madrid, Cape Town, ˜ Paulo, Delhi, Mexico City Singapore, Sao Cambridge University Press 32 Avenue of the Americas, New York, NY 10013-2473, USA www.cambridge.org Information on this title: www.cambridge.org/9781107004122 © Tomas B. Co 2013 This publication is in copyright. Subject to statutory exception and to the provisions of relevant collective licensing agreements, no reproduction of any part may take place without the written permission of Cambridge University Press. First published 2013 Printed in the United States of America A catalog record for this publication is available from the British Library. Library of Congress Cataloging in Publication Data Co, Tomas B., 1959– Methods of applied mathematics for engineers and scientists : analytical and numerical approaches / Tomas B. Co., Michigan Technological University. pages cm Includes bibliographical references and index. ISBN 978-1-107-00412-2 (hardback) 1. Matrices. 2. Differential equations – Numerical solutions. I. Title. QA188.C63 2013 512.9 434–dc23 2012043979 ISBN 978-1-107-00412-2 Hardback Additional resources for this publication at [insert URL here]. Cambridge University Press has no responsibility for the persistence or accuracy of URLs for external or third-party Internet Web sites referred to in this publication and does not guarantee that any content on such Web sites is, or will remain, accurate or appropriate.

Contents

Preface I

page xi

MATRIX THEORY

1 Matrix Algebra . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1 Definitions and Notations 4 1.2 Fundamental Matrix Operations 6 1.3 Properties of Matrix Operations 18 1.4 Block Matrix Operations 30 1.5 Matrix Calculus 31 1.6 Sparse Matrices 39 1.7 Exercises 41 2 Solution of Multiple Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 2.1 Gauss-Jordan Elimination 55 2.2 LU Decomposition 59 2.3 Direct Matrix Splitting 65 2.4 Iterative Solution Methods 66 2.5 Least-Squares Solution 71 2.6 QR Decomposition 77 2.7 Conjugate Gradient Method 78 2.8 GMRES 79 2.9 Newton’s Method 80 2.10 Enhanced Newton Methods via Line Search 82 2.11 Exercises 86 3 Matrix Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 3.1 Matrix Operators 100 3.2 Eigenvalues and Eigenvectors 107 3.3 Properties of Eigenvalues and Eigenvectors 113 3.4 Schur Triangularization and Normal Matrices 116 3.5 Diagonalization 117 3.6 Jordan Canonical Form 118 3.7 Functions of Square Matrices 120 v

vi

Contents

3.8 3.9 3.10 3.11 3.12 II

Stability of Matrix Operators Singular Value Decomposition Polar Decomposition Matrix Norms Exercises

124 127 132 135 138

VECTORS AND TENSORS

4 Vector and Tensor Algebra and Calculus . . . . . . . . . . . . . . . . . . . . 149 4.1 Notations and Fundamental Operations 150 4.2 Vector Algebra Based on Orthonormal Basis Vectors 154 4.3 Tensor Algebra 157 4.4 Matrix Representation of Vectors and Tensors 162 4.5 Differential Operations for Vector Functions of One Variable 164 4.6 Application to Position Vectors 165 4.7 Differential Operations for Vector Fields 169 4.8 Curvilinear Coordinate System: Cylindrical and Spherical 184 4.9 Orthogonal Curvilinear Coordinates 189 4.10 Exercises 196 5 Vector Integral Theorems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204 5.1 Green’s Lemma 205 5.2 Divergence Theorem 208 5.3 Stokes’ Theorem and Path Independence 210 5.4 Applications 215 5.5 Leibnitz Derivative Formula 224 5.6 Exercises 225 III

ORDINARY DIFFERENTIAL EQUATIONS

6 Analytical Solutions of Ordinary Differential Equations . . . . . . . . . . 235 6.1 First-Order Ordinary Differential Equations 236 6.2 Separable Forms via Similarity Transformations 238 6.3 Exact Differential Equations via Integrating Factors 242 6.4 Second-Order Ordinary Differential Equations 245 6.5 Multiple Differential Equations 250 6.6 Decoupled System Descriptions via Diagonalization 258 6.7 Laplace Transform Methods 262 6.8 Exercises 263 7 Numerical Solution of Initial and Boundary Value Problems . . . . . . . 273 7.1 Euler Methods 274 7.2 Runge Kutta Methods 276 7.3 Multistep Methods 282 7.4 Difference Equations and Stability 291 7.5 Boundary Value Problems 299 7.6 Differential Algebraic Equations 303 7.7 Exercises 305

Contents

8 Qualitative Analysis of Ordinary Differential Equations . . . . . . . . . . 311 8.1 Existence and Uniqueness 312 8.2 Autonomous Systems and Equilibrium Points 313 8.3 Integral Curves, Phase Space, Flows, and Trajectories 314 8.4 Lyapunov and Asymptotic Stability 317 8.5 Phase-Plane Analysis of Linear Second-Order Autonomous Systems 321 8.6 Linearization Around Equilibrium Points 327 8.7 Method of Lyapunov Functions 330 8.8 Limit Cycles 332 8.9 Bifurcation Analysis 340 8.10 Exercises 340 9 Series Solutions of Linear Ordinary Differential Equations . . . . . . . . 347 9.1 Power Series Solutions 347 9.2 Legendre Equations 358 9.3 Bessel Equations 363 9.4 Properties and Identities of Bessel Functions and Modified Bessel Functions 369 9.5 Exercises 371 IV

PARTIAL DIFFERENTIAL EQUATIONS

10 First-Order Partial Differential Equations and the Method of Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 379 10.1 The Method of Characteristics 380 10.2 Alternate Forms and General Solutions 387 10.3 The Lagrange-Charpit Method 389 10.4 Classification Based on Principal Parts 393 10.5 Hyperbolic Systems of Equations 397 10.6 Exercises 399 11 Linear Partial Differential Equations . . . . . . . . . . . . . . . . . . . . . . 405 11.1 Linear Partial Differential Operator 406 11.2 Reducible Linear Partial Differential Equations 408 11.3 Method of Separation of Variables 411 11.4 Nonhomogeneous Partial Differential Equations 431 11.5 Similarity Transformations 439 11.6 Exercises 443 12 Integral Transform Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . 450 12.1 General Integral Transforms 451 12.2 Fourier Transforms 452 12.3 Solution of PDEs Using Fourier Transforms 459 12.4 Laplace Transforms 464 12.5 Solution of PDEs Using Laplace Transforms 474 12.6 Method of Images 476 12.7 Exercises 477

vii

viii

Contents

13 Finite Difference Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 483 13.1 Finite Difference Approximations 484 13.2 Time-Independent Equations 491 13.3 Time-Dependent Equations 504 13.4 Stability Analysis 512 13.5 Exercises 519 14 Method of Finite Elements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 523 14.1 The Weak Form 524 14.2 Triangular Finite Elements 527 14.3 Assembly of Finite Elements 533 14.4 Mesh Generation 539 14.5 Summary of Finite Element Method 541 14.6 Axisymmetric Case 546 14.7 Time-Dependent Systems 547 14.8 Exercises 552 Bibliography Index

B-1 I-1

A Additional Details and Fortification for Chapter 1 . . . . . . . . . . . . . . 561 A.1 Matrix Classes and Special Matrices 561 A.2 Motivation for Matrix Operations from Solution of Equations 568 A.3 Taylor Series Expansion 572 A.4 Proofs for Lemma and Theorems of Chapter 1 576 A.5 Positive Definite Matrices 586 B Additional Details and Fortification for Chapter 2 . . . . . . . . . . . . . 589 B.1 Gauss Jordan Elimination Algorithm 589 B.2 SVD to Determine Gauss-Jordan Matrices Q and W 594 B.3 Boolean Matrices and Reducible Matrices 595 B.4 Reduction of Matrix Bandwidth 600 B.5 Block LU Decomposition 602 B.6 Matrix Splitting: Diakoptic Method and Schur Complement Method 605 B.7 Linear Vector Algebra: Fundamental Concepts 611 B.8 Determination of Linear Independence of Functions 614 B.9 Gram-Schmidt Orthogonalization 616 B.10 Proofs for Lemma and Theorems in Chapter 2 617 B.11 Conjugate Gradient Algorithm 620 B.12 GMRES algorithm 629 B.13 Enhanced-Newton Using Double-Dogleg Method 635 B.14 Nonlinear Least Squares via Levenberg-Marquardt 639 C Additional Details and Fortification for Chapter 3 . . . . . . . . . . . . . . 644 C.1 Proofs of Lemmas and Theorems of Chapter 3 644 C.2 QR Method for Eigenvalue Calculations 649 C.3 Calculations for the Jordan Decomposition 655

Contents

C.4 Schur Triangularization and SVD C.5 Sylvester’s Matrix Theorem C.6 Danilevskii Method for Characteristic Polynomial

ix

658 659 660

D Additional Details and Fortification for Chapter 4 . . . . . . . . . . . . . . 664 D.1 Proofs of Identities of Differential Operators 664 D.2 Derivation of Formulas in Cylindrical Coordinates 666 D.3 Derivation of Formulas in Spherical Coordinates 669 E Additional Details and Fortification for Chapter 5 . . . . . . . . . . . . . . 673 E.1 Line Integrals 673 E.2 Surface Integrals 678 E.3 Volume Integrals 684 E.4 Gauss-Legendre Quadrature 687 E.5 Proofs of Integral Theorems 691 F Additional Details and Fortification for Chapter 6 . . . . . . . . . . . . . . 700 F.1 Supplemental Methods for Solving First-Order ODEs 700 F.2 Singular Solutions 703 F.3 Finite Series Solution of dx/dt = Ax + b(t) 705 F.4 Proof for Lemmas and Theorems in Chapter 6 708 G Additional Details and Fortification for Chapter 7 . . . . . . . . . . . . . . 715 G.1 Differential Equation Solvers in MATLAB 715 G.2 Derivation of Fourth-Order Runge Kutta Method 718 G.3 Adams-Bashforth Parameters 722 G.4 Variable Step Sizes for BDF 723 G.5 Error Control by Varying Step Size 724 G.6 Proof of Solution of Difference Equation, Theorem 7.1 730 G.7 Nonlinear Boundary Value Problems 731 G.8 Ricatti Equation Method 734 H Additional Details and Fortification for Chapter 8 . . . . . . . . . . . . . . 738 H.1 Bifurcation Analysis 738 I Additional Details and Fortification for Chapter 9 . . . . . . . . . . . . . . 745 I.1 Details on Series Solution of Second-Order Systems 745 I.2 Method of Order Reduction 748 I.3 Examples of Solution of Regular Singular Points 750 I.4 Series Solution of Legendre Equations 753 I.5 Series Solution of Bessel Equations 757 I.6 Proofs for Lemmas and Theorems in Chapter 9 761 J Additional Details and Fortification for Chapter 10 . . . . . . . . . . . . . 771 J.1 Shocks and Rarefaction 771 J.2 Classification of Second-Order Semilinear Equations: n > 2 781 J.3 Classification of High-Order Semilinear Equations 784

x

Contents

K Additional Details and Fortification for Chapter 11 . . . . . . . . . . . . . 786 K.1 d’Alembert Solutions 786 K.2 Proofs of Lemmas and Theorems in Chapter 11 791 L Additional Details and Fortification for Chapter 12 . . . . . . . . . . . . . 795 L.1 The Fast Fourier Transform 795 L.2 Integration of Complex Functions 799 L.3 Dirichlet Conditions and the Fourier Integral Theorem 819 L.4 Brief Introduction to Distribution Theory and Delta Distributions 820 L.5 Tempered Distributions and Fourier Transforms 830 L.6 Supplemental Lemmas, Theorems, and Proofs 836 L.7 More Examples of Laplace Transform Solutions 840 L.8 Proofs of Theorems Used in Distribution Theory 846 M Additional Details and Fortification for Chapter 13 . . . . . . . . . . . . . 851 M.1 Method of Undetermined Coefficients for Finite Difference Approximation of Mixed Partial Derivative 851 M.2 Finite Difference Formulas for 3D Cases 852 M.3 Finite Difference Solutions of Linear Hyperbolic Equations 855 M.4 Alternating Direction Implicit (ADI) Schemes 863 N Additional Details and Fortification for Chapter 14 . . . . . . . . . . . . . 867 N.1 Convex Hull Algorithm 867 N.2 Stabilization via Streamline-Upwind Petrov-Galerkin (SUPG) 870

Preface

This book was written as a textbook on applied mathematics for engineers and scientist, with the expressed goal of merging both analytical and numerical methods more tightly than other textbooks. The role of applied mathematics has continued to grow increasingly important with advancement of science and technology, ranging from modeling and analysis of natural phenomenon to simulation and optimization of man-made systems. With the huge and rapid advances of computing technology, larger and more complex problems can now be tackled and analyzed in a very timely fashion. In several cases, what used to require supercomputers can now be solved using personal computers. Nonetheless, as the technological tools continue to progress, it has become even more imperative that the results can be understood and interpreted clearly and correctly, as well as the need for a deeper knowledge behind the strengths and limitations of the numerical methods used. This means that we cannot forgo the analytical techniques because they continue to provide indispensable insights on the veracity and meaning of the results. The analytical tools remain to be of prime importance for basic understanding for building mathematical models and data analysis. Still, when it comes to solving large and complex problems, numerical methods are needed. The level of exposition in this book is aimed at the graduate students, advanced undergraduate students, and researchers in the engineering and science field. Thus the topics were mostly chosen to continue several topics found in most undergraduate textbooks in applied mathematics. We have focused on advanced concepts and implementation of various mathematical tools to solve the problems that most graduate students are likely to face in their research work and other advanced courses. The contents of the book can be divided into four main parts: matrix theory, vector and tensors, ordinary differential equations, and partial differential equations. We begin the book with matrix theory because the tools developed in matrix theory form the crucial foundations used in the rest of the book. The next part centers on the concepts used in vector and tensor theory, including the application of tensor calculus and integral theorems to develop mathematical models of physical systems, often resulting in several differential equations. The last two parts focus on the solution of ordinary and partial differential equations. It can be argued that the primary needs of applied mathematics in engineering and the physical sciences are to obtain models for a system or phenomena in the form of differential equations xi

xii

Preface

and then to be able to “solve” them to predict and understand the effects of changes in model parameters, boundary conditions, or initial conditions. Although the methods of applied mathematics are independent of computing platform and programs, we have chosen to use MATLAB as a particular platform under which we investigate the mathematical methods, techniques, and ideas so that the approaches can be tested and the results can be visualized. The supplied MATLAB codes are all included on the book’s website, and the reader can modify the codes for their own use. There exists several excellent MATLAB toolboxes supplied by third-party software developers, and they have been optimized for speed, efficiency, and user-friendliness. However, the unintended consequences of user-friendly tools can sometimes render the users to be “button pushers.” We contend that students in applied mathematics still need to discover the mechanism and ideas behind the full-blown programs – at least to apply them to simple test problems and gain some basic understanding of the various approaches. The links to the supplemental MATLAB programs and files can be accessed through the link: www.cambridge.org/Co. The appendices are collected as chapter fortifications. They include proofs, advanced topics, additional tables, and examples. The reader should be able to access these materials through the web via the link: www.cambridge.org/Co. The index also contains topics that can be found in the appendices, and they are given page numbers that continue the count from the main text. Several colleagues and students have helped tremendously in the writing of this textbook. Mostly, I want to thank my best friend and wife, Faith Morrison, for the support, encouragement, and sacrifices she has given me to finish this extended and personally significant project. I hope the textbook will contain useful information to the readers, enough for them to share in the continued exploration of the methods and applications of mathematics to further improve the understanding and conditions of our world. T. Co Houghton, MI

PART I

MATRIX THEORY

Matrix theory is a powerful field of mathematics that has found applications in the solution of several real-world problems, ranging from the solution of algebraic equations to the solution of differential equations. Its importance has also been enhanced by the rapid development of several computer programs that have improved the efficiency of matrix analysis and the solution of matrix equations. We have allotted three chapters to discussing matrix theory. Chapter 1 contains the basic notations and operations. These include conventions and notations for the various structural, algebraic, differential, and integral operations. As such, this chapter focuses on how to formulate problems in terms of matrix equations, the various approaches of matrix algebraic manipulations, and matrix partitions. Chapter 2 then focuses on the solution of the linear equation given by Ax = b, and it includes both direct and indirect methods. The most direct method is to find the inverse of A and then evaluate x = A−1 b. However, the major practical issue is that matrix inverses become unwieldy when the matrices are large. This chapter is concerned with finding the solutions by reformulating the problem to take advantage of available matrix properties. Direct methods use various factorizations of A based on matrices that are more easily invertible, whereas indirect methods use an iterative process starting with an initial guess of the solution. The methods can then be applied to linear least-squares problems, as well as to the solution of multivariable nonlinear equations. Chapter 3 focuses on matrices as operators. In this case, the discussion is concerned with the analysis of matrices, for example, using eigenvalues and eigenvectors. This allows one to obtain diagonalized matrices or Jordan canonical forms. These forms provide efficient tools for evaluating matrix functions, which are also very useful for solving simultaneous differential equations. Other analysis tools such as singular values decomposition, matrix norms, and condition numbers are also included in the chapter. The matrix theory topics are also used in the other parts of this book. In Part II, we can use matrices to represent vector coordinates and tensors. The operations and vector/tensor properties can also be evaluated and analyzed efficiently using matrix theory. For instance, the mutual orthogonalities among the principal axes of a symmetric tensor are immediate consequences of the properties of matrix eigenvectors. In Part III, matrices are also shown to be indispensable tools for solving ordinary differential equations. Specifically, the solution and analysis of a set of simultaneous 1

2

Matrix Theory

linear ordinary differential equations can be represented in terms of matrix exponential functions. Moreover, numerical solution methods can now be coded in matrix forms. Finally, in Part IV of the book, both the finite difference and finite elements methods reduce partial differential equations to linear algebraic equations. Thus the tools discussed in Chapter 2 are strongly applicable because the matrices resulting from either of these methods will likely be large and sparse.

1

Matrix Algebra

In this chapter, we review some definitions and operations of matrices. Matrices play very important roles in the computation and analysis of several mathematical problems. They allow for compact notations of large sets of linear algebraic equations. Various matrix operations such as addition, multiplication, and inverses can be combined to find the required solutions in a more tractable manner. The existence of several software tools, such as MATLAB, have also made it very efficient to approach the solution by posing several problems in the form of matrix equations. Moreover, the matrices possess internal properties such as determinant, rank, trace, eigenvalues, and eigenvectors, which can help characterize the systems under consideration. We begin with the basic notation and definitions in Section 1.1. The matrix notations introduced in this chapter are used throughout the book. Then in Section 1.2, we discuss the various matrix operations. Several matrix operations should be familiar to most readers, but some may not be as familiar, such as Kronecker products. We have classified the operations as either structural or algebraic. The structural operations are those operations that involve only the collection and arrangement of the elements. On the other hand, the algebraic operations pertain to those in which algebraic operations are implemented among the elements of a matrix or group of matrices. The properties of the different matrix operations such as associativity, commutativity, and distributivity properties are summarized in Section 1.3. In addition, we discuss the properties of determinants and include some matrix inverse formulas. The properties and formulas allow for the manipulation and simplification of matrix equations. These will be important tools used throughout this book. In Section 1.4, we explore various block matrix operations. These operations are very useful when the structure of the matrices can be partitioned into submatrices. These block operations will also prove to be very useful when solving large sets of equations that exhibit a specific pattern. From algebraic operations, we then move to topics involving differential and integral calculus in Section 1.5. We first define and fix various notations for the derivatives and integrals of matrices. These notations are also used throughout the book. The various properties of the matrix calculus operations are also summarized in this section. One of the applications of matrix calculus is optimization, in which the concept of positive (and negative) definiteness is needed for sufficient conditions. We 3

4

Matrix Algebra

devote Section A.5 in the appendix to explaining positive or negative definiteness in more detail. Finally, in Section 1.6, we include a brief discussion on sparse matrices. These matrices often result when the problem involves a large collection of smaller elements that are connected with only few of the other elements, such as when we solve differential equations by numerical methods, for example, the finite difference methods or finite element methods.

1.1 Definitions and Notations The primary application of matrices is in solving simultaneous linear equations. These equations can come from solving problems based on mass and energy balance of physical, chemical, and biological processes; Kirchhoff’s laws in electric circuits; force and moment balances in engineering structures; and so forth. The size of the unknowns for these problems can be quite large, so the solution can become quite complicated. This is especially the case with modern engineering systems, which typically contain several stages (e.g., staged operations in chemical engineering), are highly integrated (e.g., large-scale integration in microelectronics), or are structurally large (e.g., large power grids and large buildings). Matrix methods offer techniques that allow for tractability and computational efficiency. When solving large nonlinear problems, numerical methods become a necessary approach. The numerical computations often involve matrix formulations. For instance, several techniques for solving nonlinear equations and nonlinear optimization problems implement Newton’s method and other gradient-based methods, in which the calculations include matrix operations. Matrix equations also result from finite approximations of systems of differential equations. For boundary value problems, the internal values are to be solved such that both the boundary conditions and the differential equations that describe the systems are satisfied. Here, the numerical techniques include finite element methods and finite difference methods, both of which translate the problem back to a linear set of equations. Aside from calculating the unknowns or solving differential equations, matrix methods are also useful in operator analysis and design. In this case, matrix equations are analyzed in terms of operators, inputs, and outputs. The matrices associated with the operators can be formulated to obtain the desired behavior. For example, if we want to move a 3D point a = (x, y, z) to another position, say, b = ( x, y, z), in a particular way, for instance, to move it radially outward or rotate it at specified degrees counterclockwise, then we can build matrices that would produce the desired effects. Conversely, for a system (mechanical, chemical, electrical, biological, etc.) that can be written in matrix forms (both in differential equations and algebraic equations), we can often isolate the matrices associated with system operations and use matrix analysis to explore the capabilities and behavior of the system. It is also worth mentioning that, in addition to the classical systems that are modeled with algebraic and differential equations, there are other application domains that use matrix methods extensively. These include data processing, computational geometry, and network analysis. In data processing, matrix methods help in regression analysis and statistical data analysis. These applications also include data mining in search engines, bioinformatics, and computer security. Computational geometry also uses matrix methods to handle and analyze large sets of data. Applications include computer graphics and visualization, which are also used for pattern

1.1 Definitions and Notations

5

recognition purposes. In network analysis, matrix methods are used together with graph theory to analyze the connectivity and effects of large, complex structures. Applications include the analysis of communication and control systems, as well as large power grids. We now begin with the definition of a matrix and continue with some of the notations and conventions that are used throughout this book. Definition 1.1. A matrix is a collection of objects, called the elements of the matrix, arranged in rows and columns. These elements of the matrix could be numbers, such as   √ 1 0 −0.3 A= with i = −1 1 −2 3 + i 2 or functions, such as

 B=



2x(t) + a dy/dt

1 sin(ωt)dt



The elements of matrices are restricted to a set of mathematical objects that allow algebraic binary operations such as addition, subtraction, multiplication, and division. The valid elements of the matrix are referred to as scalars. Note that a scalar is not the same as a matrix having only one row and one column. We often use capital letters to denote matrices, whereas the corresponding small letters stand for the elements. Thus the elements of matrix A positioned at the ith row and j th column are denoted as aij , for example, for A having N rows and M columns, ⎛ ⎞ a11 a12 · · · a1M ⎜ a21 a22 · · · a2M ⎟ ⎜ ⎟ (1.1) A=⎜ . .. .. ⎟ .. ⎝ .. . . . ⎠ aN1

aN2

···

aNM

The size of the matrix is given by the symbol “[=]”, for example, for matrix A having N rows and M columns, A [=] N × M

or

A[N×M]

(1.2)

A row vector is a matrix having one row, whereas a column vector is a matrix having one column. The length of a vector means the number of elements of the row or column vector. If the type of vector has not been specified, we take it to mean a column vector. We often use bold small letters to denote vectors. A basic vector is the ith unit vector of length N denoted by ei , ⎞ ⎛ 0 ⎜ .. ⎟ ⎜ . ⎟ ⎟ ⎜ ⎜ 0 ⎟ ⎟ ⎜ th ⎟ ei = ⎜ (1.3) ⎜ 1 ⎟ ← i element ⎜ 0 ⎟ ⎟ ⎜ ⎜ . ⎟ ⎝ .. ⎠ 0 The length N of the unit vector is determined by context.

6

Matrix Algebra

A square matrix is a matrix with the same number of columns and rows. Special cases include lower triangular, upper triangular, and diagonal matrices. Lower triangular matrices have zero elements above the main diagonal, whereas upper triangular matrices have zero elements below the main diagonal. Diagonal matrices have zero off-diagonal elements. The diagonal matrix is also represented by   D = diag d11 , d22 , . . . , dNN

(1.4)

A special diagonal matrix in which the main diagonal elements are all 1’s is known as the identity matrix, denoted by I. If the size of the identity matrix needs to be specified, then we use IN to denote an N × N identity matrix. An extensive list of different matrices that have special forms such as bidiagonal, tridiagonal, Hessenberg, Toeplitz, and so forth are given in Tables A.1 through A.5 in Section A.1 as an appendix for easy reference.

1.2 Fundamental Matrix Operations We assume that the reader is already familiar with several matrix operations. The purpose of the following sections is to summarize these operations, introduce our notations, and relate them to some of the available MATLAB commands. We can divide matrix operations into two major categories. The first category involves the restructuring or combination of matrices. The second category includes the operations that contain algebraic computations such as addition, multiplication, and inverses.

1.2.1 Matrix Restructuring Operations A list of matrix rearrangement operations with their respective notations are summarized in Tables 1.1 and 1.2 (together with some MATLAB commands associated with the operations). The row and column augmentation operations are designated by horizontal and vertical bars, respectively. These are used extensively throughout the book because we take advantage of block matrix operations. The reshaping operations are given by the vectorization operation and reshape operation. Both these operations are quite useful when reformulating equations such as HX + XB + CXD = F into the familiar linear equation form given by Ax = b. There are two operations that involve exchanging the roles of rows and columns: the standard transpose operation, which we denote by superscript T , and the conjugate transpose, which we denote by superscript asterisk. In general, AT = A∗ , except when the elements of A are all real. When A = AT , we say that A is symmetric, and when A = A∗ , we say that A is Hermitian. The two cases are generally not the same. For instance, let  A=

1+i 2

2 3



 B=

1 2−i

2+i 3



then A is symmetric but not Hermitian, whereas B is Hermitian but not symmetric. On the other hand, when A = −AT , we say that A is skew-symmetric, and when A = −A∗ , we say that A is skew-Hermitian.

1.2 Fundamental Matrix Operations

7

Table 1.1. Matrix restructuring operations Operation

Notation

C= 1





A

B

Rule ⎛ c11 ⎜ . ⎜ . ⎝ . cN1 ⎛

Column Augment

⎜ ⎜ ⎝

MATLAB: C=[A,B]

⎛

C= 2

  A B

⎜ ⎜ ⎝

··· .. . ···

⎞ c1,M+P .. ⎟ ⎟ . ⎠= cN,M+P

a11 .. . aN1

· · · a1M .. .. . . · · · aNM

c11 .. . cN+P,1

⎞ ··· c1,M ⎟ .. .. ⎟= . ⎠ . · · · cN+P,M ⎛ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎝

Row Augment MATLAB: C=[A;B]

⎛ C = vec (A) 3

Vectorize MATLAB: C=A(:)

⎛

⎞ · · · b1P .. ⎟ .. ⎟ . . ⎠ · · · bNP

b11 .. . bN1

a11 .. . aN1 b11 .. . bP1

· · · a1M .. .. . . · · · aNM · · · b1M .. .. . . · · · bPM

⎞

⎞

⎜ A•,1 ⎟ ⎜ ⎟ c1 ⎟ ⎜ . ⎟ ⎜ .. ⎟ ⎜ . ⎟=⎜ ⎜ ⎝ . ⎠ ⎜ . ⎟ ⎟ ⎜ ⎟ cN·M ⎝ ⎠ A•,M where A•,i is the ith column of A

The submatrix operation is denoted by using a list of k subscript indices and  superscript indices to refer to the rows and columns, respectively, extracted from a matrix. For instance, ⎛ ⎞   1 2 3 2 3 = A = ⎝ 4 5 6 ⎠ → A2,3 1,2 5 6 7 8 9 For a square matrix, if the diagonals of the submatrix are a subset of the diagonals of the original matrix, then we call it a principal submatrix. This happens if the superscript indices and the subscript indices of the submatrix are the same. For instance, ⎛ ⎞   1 2 3 1 3 = A = ⎝ 4 5 6 ⎠ → A1,3 1,3 7 9 7 8 9 then A1,3 1,3 is a principal submatrix.

⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠

8

Matrix Algebra Table 1.2. Matrix rearrangement operations Operation

Notation

Rule ⎛

C= reshape (v, N, M) 4

v1 ⎜ . ⎜ C = ⎝ .. vN

Reshape MATLAB: reshape(v,N,M)

⎛ ⎜ C=⎜ ⎝

Transpose MATLAB: C=A.’

⎞ · · · aM1 .. ⎟ .. ⎟ . . ⎠ · · · aMN

a11 .. . a1M ⎛

a11 ⎜ . ⎜ C = ⎝ .. a1M

∗

C=A 6

⎞ v(M−1)N+1 ⎟ .. ⎟ ⎠ . ··· vMN

where v is a vector of length NM

C = AT 5

vN+1 .. . v2N

Conjugate Transpose

⎞ · · · aM1 .. ⎟ .. ⎟ . . ⎠ · · · aMN

MATLAB: C=A’ where aij = complex conjugate of aij j ,j ...,j

C = Ai11,i22,...,ik 7

Submatrix

⎛

MATLAB: rows=[i1,i2,...] cols=[j1,j2,...] C=A(rows,cols) C = Aij ↓

ai1 j 1 ⎜ . ⎜ C[ k× ] = ⎝ .. aik j 1

⎞ · · · ai1 j  .. ⎟ .. ⎟ . . ⎠ · · · aik j 

C[ (N−1)×(M−1) ] = ⎛

8

⎞ 1,...,j −1 A1,...,i−1

(ij )th Redact

⎜ ⎜ ⎜ ⎜ ⎜ 1,...,j −1 ⎝ A i+1,...,N

MATLAB: C=A C(i,j)=[ ]

j +1,...,M A1,...,i−1 j +1,...,M

Ai+1,...,N

⎟ ⎟ ⎟ ⎟ ⎟ ⎠

Next, the operation to remove some specified rows and columns is referred to here as the (ij )th redact operation. We use Aij ↓ to denote the removal of the ith row and j th column. For instance, ⎛

⎞ 1 2 3 A=⎝ 4 5 6 ⎠ 7 8 9

 →

A23↓ =

1 7

2 8



This operation is useful in finding determinants, cofactors, and adjugates.

(1.5)

1.2 Fundamental Matrix Operations

9

Table 1.3. Matrix algebraic operations Operation

Notation

Rule

MATLAB commands

1

Sum

C = A+B

cij = aij + bij

C=A+B

2

Scalar Product

C = qA

cij = q aij

C=q*A

3

Matrix Product

C = AB

cij =

4

Haddamard Product

C = A◦B

cij = aij bij

5

Kronecker Product (tensor product)

6

Determinant

K k=1

C=A.*B

C⎛= C = A⊗B

q = det (A) or q = |A|

···



N 

(K)

⎞ a1M B ⎟ .. ⎟ ⎠ . aNM B

···

a11 B ⎜ .. ⎜ ⎝ . aN1 B

q=

C=A*B

aik bkj

K

C=kron(A,B)

 ai,ki

i=1

q=det(A)

7

Cofactor

q = cof (aij )

see (1.10)     q = (−1)i+j  Aij ↓ 

8

Adjugate

C = adj (A)

cij = cof (a ji )

9

Inverse

C = A−1

C=

10

Trace

q = tr (A)

q=

11

Real Part

C = Real(A)

cij = real (aij )

C=Real(A)

12

Imag Part

C = Imag(A)

cij = imag (aij )

C=Imag(A)

13

Complex Congugate

C=A

cij = aij

C=Conj(A)

1 adj (A) |A|

N i=1

aii

C=inv(A) q=trace(A)

1.2.2 Matrix Algebraic Operations The matrix algebraic operations can be classified further as either binary or unary. For binary operations, the algebraic operations require two inputs, either a scalar and a matrix or two matrices of appropriate sizes. For unary operations, the input is a matrix, and the algebraic operations are applied on the elements of the matrix. The matrix algebraic operations are given in Table 1.3, together with their corresponding MATLAB commands.

10

Matrix Algebra

1.2.2.1 Binary Algebraic Operations

The most basic matrix binary computational operations are matrix sums, scalar products, and matrix products, which are quite familiar to most readers. To see how these operations seem the natural consequences of solving simultaneous equations, we refer the reader to Section A.2 included in the appendices. Matrix products of A and B, are denoted simply by C = AB, which requires A[=]N × K, B[=]K × M and C[=]N × M (i.e., the columns of A must be equal to the rows of B). If this is the case, we say that A and B are conformable for the operation AB. Furthermore, based on the sizes of the matrices, A[N×K] B[K×M] = C[N×M] , we see that dropping the common value K leaves the size of C to be N × M. For the matrix product AB, we say that A premultiplies B, or B postmultiplies A. For instance, let ⎛

⎞ 1 1 A=⎝ 2 1 ⎠ −1 0

 and

B=

−2 −1

1 3

⎛

 then

⎞ −3 4 C = AB = ⎝ −5 5 ⎠ 2 −1

However, B and A is not conformable for the product BA. In several cases, AB = BA, even if the reversed order is conformable, and thus one needs to be clear whether a matrix premultiplies or postmultiplies another matrix. For the special case in which switching the order yields the same product (i.e., AB = BA), then we say that A and B commutes. It is necessary that commuting matrices are square and have the same sizes. We list a few key results regarding matrix products: 1. For matrix products between a matrix A[=]N × M and the appropriately sized identity matrix, we have AIM = IN A = A where IM and IN are identity matrices of size M and size N, respectively. 2. Based on the definition of matrix products, when B premultiplies A, the row elements of B are pairwise multiplied with the column elements of A, and the results are then summed together. This fact implies that to scale the ith row of A by a factor di , we can simply premultiply A by a diagonal matrix D = diag (d1 , . . . , dN ). For instance, ⎛

⎞⎛ 0 1 0 ⎠⎝ 4 −1 7

2 0 DA = ⎝ 0 1 0 0

2 5 8

⎞ ⎛ 3 2 4 6 ⎠=⎝ 4 5 9 −7 −8

⎞ 6 6 ⎠ −9

Likewise, to scale the j th column of A by a factor d j , we can simply postmultiply A by a diagonal matrix D = diag (d1 , . . . , dN ). For instance, ⎛

1 AD = ⎝ 4 7

2 5 8

⎞⎛ ⎞ ⎛ 3 2 0 0 2 6 ⎠⎝ 0 1 0 ⎠=⎝ 8 9 0 0 −1 14

⎞ 2 −3 5 −6 ⎠ 8 −9

1.2 Fundamental Matrix Operations

3. Premultiplying A by a row vector of 1’s yields a row vector containing the sums of each column, whereas postmultiplying by a column vector of 1’s yields a column vector containing the sum of each row. For instance, ⎛ ⎛ ⎞ ⎞⎛ ⎞ ⎛ ⎞ 1 2 3 1 2 3 1 6     ⎝ 4 5 6 ⎠ ⎝1⎠ = ⎝15⎠ 1 1 1 ⎝ 4 5 6 ⎠ = 12 15 18 7 8 9 7 8 9 1 24 4. Let T be an identity matrix, but with additional nonzero nondiagonal elements in the j th column. Then B = TA is a matrix whose ith row (i = j ) is given by the sum of the ith row of A and tij · (j th row of A). The j th row of B remains to be the j th row of A. For instance, ⎛ ⎞⎛ ⎞ ⎛ ⎞ 1 0 −1 1 2 3 −6 −6 −6 ⎝ 0 1 2 ⎠ ⎝ 4 5 6 ⎠ = ⎝ 18 21 24 ⎠ 0 0 1 7 8 9 7 8 9 Likewise, let G be an identity matrix, but with nondiagonal elements in the ith row. Then C = AG is a matrix whose j th column (j = i) is given by the sum of the j th column of A and g ij · (j th column of A). The ith column of G remains to be the ith column of A. For instance, ⎛ ⎞⎛ ⎞ ⎛ ⎞ 1 2 3 1 0 0 −2 8 3 ⎝ 4 5 6 ⎠ ⎝ 0 1 0 ⎠ = ⎝ −2 17 6 ⎠ 7 8 9 −1 2 1 −2 26 9 5. A square matrix P is known as a row permutation matrix if it is a matrix obtained by permuting the rows of an identity matrix. If P is a row permutation matrix, then PA is a matrix obtained by permuting the rows of A in the same sequence as P. For instance, let P[=]3 × 3 be obtained by permuting the rows of the identity matrix according to the sequence [3, 1, 2], then ⎛ ⎞⎛ ⎞ ⎛ ⎞ 0 0 1 1 2 3 7 8 9 PA = ⎝ 1 0 0 ⎠ ⎝ 4 5 6 ⎠ = ⎝ 1 2 3 ⎠ 0 1 0 7 8 9 4 5 6 Likewise, a square matrix  P is also a column permutation matrix if it is a matrix obtained by permuting the columns of an identity matrix. If  P is a column permutation matrix, then A P is obtained by permuting the columns A in the same sequence as  P. For instance, let  P[=]3 × 3 be obtained by permuting the columns of the identity matrix according to the sequence [3, 1, 2], then ⎛ ⎞⎛ ⎞ ⎛ ⎞ 1 2 3 0 1 0 3 1 2 A P = ⎝ 4 5 6 ⎠⎝ 0 0 1 ⎠ = ⎝ 6 4 5 ⎠ 7 8 9 1 0 0 9 7 8 Remark: Matrices D, T , and P described in items 2, 4, and 5 are known as the scaling, pairwise combination, and permutation row operators, respectively. Collectively, they are known as the elementary row operators. All three operations show that premultiplication (left multiplication) is a row operation. On the other hand, D, G, and  P are elementary column operators, and they operate on matrices

11

12

Matrix Algebra

via postmultiplication (right multiplication).1 All these matrix operations are used extensively in the Gauss-Jordan elimination method for solving linear equations. Aside from scalar and matrix products, there are two more matrix operations involving multiplication. The Hadamard product, also known as element-wise product, is defined as follows: Q = A◦B For instance,



←→

1 −1 −2 2

qij = aij bij   1 ◦ 3

2 4

i = 1, . . . , N; j = 1, . . . , M



 =

1 −2 −6 8

(1.6)



The Kronecker product, also known as the Tensor product, is defined as follows: ⎛ ⎞ a11 B · · · a1M B ⎜ ⎟ .. .. ..  = A⊗B = ⎝ (1.7) ⎠ . . . aN1 B · · · aNM B where the matrix blocks aij B are scalar products of aij and B. For instance, ⎛ ⎞ 1 2 −1 −2     ⎜ 3 4 −3 −4 ⎟ 1 −1 1 2 ⎟ ⊗ =⎜ ⎝ −2 −4 2 4 ⎠ −2 2 3 4 −6 −8 6 8 Both the Hadamard product and Kronecker product are useful when solving general matrix equations, some of which result from the finite difference methods. 1.2.2.2 Unary Algebraic Operations

We first look at the set of unary operations applicable only to square matrices. The first set of unary operations to consider are highly related to each other. These operations are the determinant, cofactors, adjugates, and inverses. As before, we refer the reader to Section A.2 to see how these definitions naturally developed from the application to the solution of simultaneous linear algebraic equations. Of these unary operations, the matrix inverse can easily be defined independent of computation. Definition 1.2. The matrix inverse of a square matrix A is a matrix of the same size, denoted by A−1 , that satisfies A−1 A = AA−1 = I

(1.8)

Unfortunately, except for some special classes of matrices, the determination of the inverse is not straightforward in general. Instead, the computation of matrix inverses requires the definition of three other operations: the determinant, the cofactor, and the adjugate. First, we need another function called the permutation sign function. 1

We suggest the use of the mnemonics LR and RC to stand for Left operation acts on Rows and Right operation acts on Columns, respectively.

1.2 Fundamental Matrix Operations

13

Definition 1.3. Let K = {k1 , k2 , . . . , kN } be a sequence of distinct indices ranging from 1 to N. Let π(K) be the number of pairwise exchanges among the indices in the sequence K needed to reorder the sequence in K into an ascending order given by {1, 2, . . . , N}. Then the permutation sign function, denoted by (K), is defined by (K) = (−1)π(K)

(1.9)

which means it takes on the value of +1 or −1, depending on whether π(K) is even or odd, respectively. For example, we have (1, 2, 3) = +1

(5, 1, 2, 4, 3) = −1

(2, 1, 3, 4) = −1

(6, 2, 1, 5, 3, 4) = +1

Definition 1.4. The determinant of a square matrix A of size N, denoted by either |A| or det(A), is given by   (k1 , . . . , kN ) a1,k1 a2,k2 · · · aN,kN (1.10) det(A) = ki = k j i, j = 1, . . . , N where the summation is over all nonrepeated combinations of indices 1, 2, . . . , N. Definition 1.5. The cofactor of an element aij of a square matrix A of size N, denoted by cof (aij ), is defined as cof (aij ) = (−1)i+j det (Aij ↓ )

(1.11)

where Aij ↓ is the (ij )th redact. Using cofactors, we can compute the determinant in a recursive manner. LEMMA 1.1.

Let A be a square matrix of size N, then det (A) = a11 if N = 1. Otherwise,

for any j det (A) =

N 

akj cof (akj )

(1.12)

aik cof (aik )

(1.13)

k=1

Likewise, for any i det (A) =

N  k=1

By induction, one can show that either the column expansion formula given in (1.12) or the row expansion formula given in (1.13) will yield the same result as given in (1.10).

PROOF.

We refer to A as singular if det(A) = 0; otherwise, A is nonsingular. As we show next, only nonsingular matrices can have matrix inverses.

14

Matrix Algebra

Definition 1.6. The adjugate2 of a square matrix A is a matrix of the same size, denoted by adj (A), consisting of the cofactors of each element in A but collected in a transposed arrangement, that is, ⎞ ⎛ cof · · · cof (a ) (a ) 11 N1 ⎟ ⎜ ⎟   ⎜ ⎟ ⎜ . . .. .. .. (1.14) adj A = ⎜ ⎟ . ⎟ ⎜ ⎠ ⎝ cof (a1N ) · · · cof (aNN ) Using adjugates, we arrive at one key result for the computation of matrix inverses, if they exist. LEMMA 1.2.

Let A be any square matrix, then   A adj(A) = det(A) I and

  adj(A) A = det(A) I

(1.15)

Assuming matrix A is nonsingular, the inverse is given by A−1 =

1 adj(A) det(A)

(1.16)

(See section A.4.3) Note that matrix adjugates always exist, whereas matrix inverses A−1 exist only if det(A) = 0.

PROOF.

EXAMPLE 1.1.

Let

⎛

1 A=⎝ 4 7 then

then

⎞ 3 6 ⎠ 0

 5 cof(a11 ) = +  8

 6 ; 0

   2 3 ;  cof(a21 ) = −  8 0

  2 3   cof(a31 ) = +  5 6

 4 cof(a12 ) = −  7

 6 ; 0

   1 3 ;  cof(a22 ) = +  7 0

  1 3   cof(a32 ) = −  4 6

 4 cof(a13 ) = +  7

 5 ; 8

   1 2 ;  cof(a23 ) = −  7 8

  1 2   cof(a33 ) = +  4 5

⎛

−48 adj(A) = ⎝ 42 −3 2

2 5 8

24 −21 6

⎞ ⎛ −3 27 6 ⎠ → adj(A) A = A adj(A) ⎝ 0 −3 0

⎞ 0 0 27 0 ⎠ 0 27

In other texts, the term “adjoint” is used instead of adjugate. We chose to use the latter, because the term “adjoint” is also used to refer to another matrix in linear operator theory.

1.2 Fundamental Matrix Operations

and A−1

⎛ −48 1 ⎝ = 42 27 −3

15

⎞ 24 −3 −21 6 ⎠ 6 −3

Although (1.16) is a general method for computing the inverse, there are more efficient ways to find the matrix inverse that take advantage of special structures and properties. For instance, the inverse of diagonal matrices is another diagonal matrix consisting of the reciprocals of the diagonal elements. Another example is when the transpose happens to also be its inverse. These matrices are known as orthogonal matrices. To determine whether a given matrix is indeed orthogonal, we can just compute AT A and AAT and check whether both products yield identity matrices. The other unary operations include the trace, real component, imaginary component, and the complex conjugate operations. The trace of a square matrix A, denoted tr(A), is defined as the sum of the diagonals. Let A[=]2 × 2, then for M = λI − A, where λ is a scalar parameter, we have the following results:     det (λI − A) = λ2 − tr (A) λ + det A   λ − a22 −a12 adj (λI − A) = −a21 λ − a11   1 λ − a22 −a12 −1     = (λI − A) −a21 λ − a11 λ2 − tr (A) λ + det A

EXAMPLE 1.2.

Note that when det (λI − A) = 0, the inverse will no longer exist, but adj (λI − A) will still be valid. We now show some examples in which the matrices can be used to represent the indexed equations. The first example involves the matrix formulation of the finite difference approximation of a partial differential equation. The second involves the matrix formulation of a quadratic equation. EXAMPLE 1.3.

by

Consider the heat equation of a L × W flat rectangular plate given  2  ∂ T ∂T ∂2T =α + ∂t ∂x2 ∂y2

(1.17)

with stationary boundary conditions T (0, y, t) = f 0 (y) T (L, y, t) = f L(y)

T (x, 0, t) = g 0 (x) T (x, W, t) = g W (x)

and initial condition, T (x, y, 0) = h(x, y). We can introduce a uniform finite time increment t and finite differences for x and y given by x = L/(N + 1) and y = W/(M + 1), respectively, so that tk = k t, xn = n x, and ym = m y,

16

Matrix Algebra Tn-1,m Tn,m-1

Tn,m+1 Tn,m

y=0 x=0

y

Tn+1,m

...

x

T(k+1) T(k)

...

L

T(0)

W

Figure 1.1. A schematic of the finite difference approximation of the temperature distribution T of a flat plate in Example 1.3.

with k = 0, 1, . . ., n = 0, . . . , N + 1, and m = 0, . . . , M + 1. The points corresponding to n = 0, n = N + 1, m = 0, and m = M + 1 represent the boundary values. We can then let [T (k)] be a N × M matrix that represents the temperature distribution of the specific internal points of the plate at time tk (see Figure 1.1). Using the finite difference approximation of the partial derivatives at point x = n x and y = m y, and time t = k t:3 ∂T ∂t

=

T n,m (k + 1) − T n,m (k)

t

∂2T ∂x2

=

T n+1,m (k) − 2T n,m (k) + T n−1,m (k)

x2

∂2T ∂y2

=

T n,m+1 (k) − 2T n,m (k) + T n,m−1 (k)

y2

then (1.17) is approximated by the following indexed equations:     1 μx T n−1,m (k) + − 2 T n,m (k) + T n+1,m (k) 2μx   T n,m (k + 1) =   1 + μy T n,m−1 (k) + − 2 T n,m (k) + T n,m+1 (k) 2μy

(1.18)

where μx =

α t ( x)2

;

μy =

α t ( y)2

T n,m (k) is the temperature at time t = k t located at (x, y) = (n x, m y). The first group of terms in (1.18) involves only T n−1,m , T n,m , and T n+1,m , that is, only a combination of row elements at fixed m. This means that the 3

The finite difference methods are discussed in more detail in Chapter 13.

1.2 Fundamental Matrix Operations

17

first group of terms can be described by the product AT for some constant N × N matrix A. Conversely, the second group of terms in (1.18) involves only acombination of column elements at fixed n, which means a product TB for some matrix B[=]M × M. In anticipation of boundary conditions, we need an extra matrix C[=]N × M. Thus we should be able to represent (1.18) using a matrix formulation given by T (k + 1) = AT (k) + T (k)B + C

(1.19)

4

where A and B and C are constant matrices. When formulating general matrix equations, it is often advisable to apply it to smaller matrices first. Thus let us start with a case in which N = 4 and M = 3. We can show that (1.18) can be represented by ⎞ ⎛ ⎞⎛ T 11 (k) T 12 (k) T 13 (k) βx 1 0 0 ⎜ ⎟ ⎜1 β   1 0⎟ x ⎜ ⎟ ⎜T 21 (k) T 22 (k) T 23 (k)⎟ T (k + 1) = μx ⎜ ⎟ ⎟⎜ ⎝ 0 1 βx 1 ⎠ ⎝T 31 (k) T 32 (k) T 33 (k)⎠ 0 0 1 βx T 41 (k) T 42 (k) T 43 (k) ⎞ ⎛ ⎞ T 11 (k) T 12 (k) T 13 (k) ⎛ ⎜T (k) T (k) T (k)⎟ βy 1 0 22 23 ⎟⎜ ⎜ 21 ⎟ + μy ⎜ ⎟ ⎝ 1 βy 1 ⎠ ⎝T 31 (k) T 32 (k) T 33 (k)⎠ 0 1 βy T 41 (k) T 42 (k) T 43 (k) ⎞ ⎞ ⎛ ⎛ T 01 (k) T 02 (k) T 03 (k) T 10 (k) 0 T 14 (k) ⎜ 0 ⎜T (k) 0 T (k)⎟ 0 0 ⎟ 24 ⎟ ⎟ ⎜ ⎜ 20 + μx ⎜ ⎟ + μy ⎜ ⎟ ⎝ 0 ⎝T 30 (k) 0 T 34 (k)⎠ 0 0 ⎠ T 51 (k) T 52 (k) T 53 (k) T 40 (k) 0 T 44 (k) where βx = 1/(2μx ) − 2 and βy = 1/(2μy ) − 2. Generalizing, we have ⎛ ⎞ βx 1 0 ⎜ ⎟ ⎜ 1 ... ... ⎟ ⎜ ⎟ [=]N × N A = μx ⎜ ⎟ .. ⎝ . βx 1 ⎠ 0 1 βx ⎛ ⎞ βy 1 0 ⎜ ⎟ ⎜ 1 ... ... ⎟ ⎟ [=]M × M B = μy ⎜ ⎜ ⎟ .. ⎝ . βy 1 ⎠ 0 ⎛

C

4

=

p1 ⎜ 0 ⎜ ⎜ μx ⎜ ⎜ ⎝ 0 q1

1 ··· ··· .. . ··· ···

βy ⎞

pM ⎛ 0 ⎟ ⎟ ⎟ ⎜ ⎟ + μy ⎝ ⎟ 0 ⎠ qM

r1 .. . rN

0 .. . ··· 0

0 .. . 0

⎞ s1 .. ⎟ . ⎠ sN

More generally, if the boundary conditions are time-varying, then C = C(k). Also, if the coefficient α = α(t), then A and B will need to be replaced by A(k) and B(k), respectively.

18

Matrix Algebra

where p m = f 0 (m y), qm = f L (m y), rn = g 0 (n x), and sn = g W (n x). The initial matrix is obtained using the initial condition, that is, T nm (0) = h(n x, m y). Starting with T (0), one can then march iteratively through time using (1.19). (A specific example is given in exercise E1.21.)

EXAMPLE 1.4.

The general second-order polynomial equation in N variables is

given by β=

N  N 

aij xi x j

i=1 j =1

One could write this equation as [β] = xT Ax where

⎛

a11 ⎜ A = ⎝ ... aN1

... .. . ...

⎞ ⎛ ⎞ a1N x1 .. ⎟ and x = ⎜ .. ⎟ ⎝ . ⎠ . ⎠ aNN xN

Note that [β] is a 1 × 1 matrix in this formulation. The right-hand side is known as the quadratic form. However, because xi x j = x j xi , three alternative forms are possible: [β] = xT Qx

[β] = xT Lx

or

[β] = xT Ux

where ⎛

⎞ ⎛ ⎞ ⎛ ⎞ 0 u11 . . . u1N 11 q11 . . . q1N ⎟ ⎜ .. ⎟ L = ⎜ .. .. ⎟ U = ⎜ .. .. .. Q = ⎝ ... ⎝ ⎝ . ⎠ . . . . ⎠ . ⎠ N1 . . . NN 0 uNN qN1 . . . qNN and aij + a ji qij = ; 2

⎧ ⎨ aij + a ji uij = a ⎩ ii 0

if i < j if i = j ; if i > j

⎧ ⎨ aij + a ji ij = a ⎩ ii 0

if i > j if i = j if i < j

(The proof that all three forms are equivalent is left as an exercise in E1.34.) This example shows that more than one matrix formulation is possible in some cases. Matrix Q is symmetric, whereas L is lower triangular, and U is upper triangular. The most common formulation is to use the symmetric matrix Q.

1.3 Properties of Matrix Operations In this section, we discuss the different properties of matrix operations. With these properties, one could manipulate matrix equations to either simplify equations, generate efficient algorithms, or analyze the problem, before actual matrix computations. We first discuss the basic properties involving addition, multiplications, and

1.3 Properties of Matrix Operations Table 1.4. Properties of matrix operations Commutative Operations A◦B αA

= =

B◦A Aα

A+B

=

B+A

−1

=

A−1 A

AA

Associativity of Sums and Products A + (B + C)

=

(A + B) + C

A (BC)

=

(AB) C

A ◦ (B ◦ C)

=

(A ◦ B) ◦ C

A ⊗ (B ⊗ C)

=

(A ⊗ B) ⊗ C

Distributivity of Products A (B + C)

=

AB + AC

A ⊗ (B + C)

=

A⊗B+A⊗C

(A + B) C

=

AC + BC

(A + B) ⊗ C

=

A⊗C+B⊗C

A ◦ (B + C)

=

A◦B+A◦C

= =

B◦A+C◦A (B + C) ◦ A

(AB) ⊗ (CD) = (A ⊗ C)(B ⊗ D)

Transpose of Products (AB)T

=

BT AT

(A ⊗ B)T

=

AT ⊗ BT

(A ◦ B)T

= =

BT ◦ AT AT ◦ BT

Inverse of Matrix Products and Kronecker Products (AB)−1 = B−1 A−1

(A ⊗ B)−1 = (A)−1 ⊗ (B)−1 Reversible Operations

 T T A ∗ ∗

(A )

=

A

=

A

 −1 −1 =A A

Vectorization of Sums and Products vec (A + B) = vec (A) + vec (B)

vec (BAC)

=

 T  C ⊗ B vec (A)

vec (A ◦ B)

=

vec(A) ◦ vec(B)

inverses. Next is a separate subsection on the properties of determinants. Finally, we include a subsection of the formulas that involve matrix inverses.

1.3.1 Basic Properties A list of some basic properties of matrix operations is given in Table 1.4. Most of the properties can be derived by directly using the definitions given in Tables 1.1, 1.2, and 1.3. The proofs are given in Section A.4.1 as an appendix. The properties of the matrix operations allow for the manipulation of matrix equations before

19

20

Matrix Algebra Table 1.5. Definition of vectors Vector

Description of elements

x y z w

xk is the annual supply rate (kg/year) of material from source k. yk is the annual production rate (kg/year) of product k. zk is the sale price per kg of product k. wk is the production cost per kg of the material from source k.

actual computations. They help in simplifying expressions that often yield important insights about the data or the system being investigated. The first group of properties list the commutativity, associativity, and distributivity properties of various sums and products. One general rule is to choose associations of products that would improve computations. For instance, let a, b, c, d, e, and f be column vectors of the same length; we should use the following associations    abT cdT ef T = a bT c dT e f T     because both bT c and dT e are 1 × 1. A similar rule holds for using the distributive properties. For example, we can use distributivity to rearrange the following equation: AD + ABCD = A(D + BCD) = A(I + BC)D More importantly, these properties allow for manipulations of matrix equations to help simplify the equations, as shown in the example that follows. Consider a processing facility that can take raw material from M different sources to produce N different products. The fractional yield of product j per kilogram of material coming from source i can be collected in matrix form as F = ( f ij ). In addition, define the cost, price, supply rates, and production rates by the column vectors given in Table 1.5. We simplify the situation by assuming that all the products are sold immediately after production without need for inventory. Let S, C, and P = (S − C) be the annual sale, annual cost, and annual net profit, respectively. We want to obtain a vector g where the kth element is the annual net profit per kilogram of material from source k, that is, P = gT x. Using matrix representation, we have EXAMPLE 1.5.

y

=

Fx

S

=

zT y

C

=

wT x

then the net profit can be represented by

  P = S − C = zT F x − wT x = zT F − wT x = gT x

where g is given by g = FTz − w

1.3 Properties of Matrix Operations

More generally, the problem of maximizing net profit by adjusting the supply rates are formulated as a typical linear programming problem: max x

gT x

(objective function)

subject to 0≤ ymin ≤

x

≤ xmax

(availability constraints)

y(= F x)

≤ ymax

(demand constraints)

The transposes of matrix products turn out to be equal to the matrix products of the transposes but in the reversed sequence. Together with the associative property, this can be extended to the following results: (ABC · · · EFG)T  T Ak  −1 T AA

= =

GT F T ET · · · CT DT AT  T k A

=

 −1 T T  −1 T A A = I = AT A

 −1  −1 T The last result shows that AT = A . Thus we often use the shorthand A−T  T −1  −1 T to mean either A or A . Similarly, the inverse of a matrix product is a product of the matrix inverses in the reverse sequence. This can be generalized to be5 (ABC · · ·)−1  −1 Ak

=

· · · C−1 B−1 A−1

=

k  A−1 · · · A−1 = A−1

Ak A

=

Ak+

k  −1  Thus we can use A−k to denote either Ak or A−1 . Note that these results are still consistent with A0 = I. Consider a resistive electrical network consisting of junction points or nodes that are connected to each other by links where the links contain three types of electrical components: resistors, current sources, and voltage sources. We simplify our network to contain only two types of links. One type of link contains only either one resistor or one voltage source, or both connectedin series.6 EXAMPLE 1.6.

5

Note that this is not the case for Kronecker products that is, (A ⊗ B ⊗ C ⊗ · · ·)−1 = A−1 ⊗ B−1 ⊗ C−1 ⊗ · · ·

6

If multiple resistors with resistance Rj 1 , Rj 2 , . . . are connected in series the j th link, then they can  be replaced by one resistor with resistance Rj = k Rjk . Likewise, if multiple voltage sources with signed voltages s j 1 , s j 2 , . . . are connected in series the j th link, then they can be replaced by one  voltage source with signed voltage s j = k s jk , where the sign is positive if the polarity goes from positive to negative along the current flow.

21

22

Matrix Algebra R2 1

2 R4

R1 + S1 -

R5

R3 R6

0

Figure 1.2. An electrical network with resistors Rj in link j , voltage sources s j in link j , and current sources Ak, from node k to node .

3 A3,0

The other type of link contains only a current source. One such network is shown in Figure 1.2. Suppose there are n + 1 nodes and m ≥ (n + 1) links. By setting one of the nodes as having zero potential (the ground node), we want to determine the potentials of the remaining n nodes as well as the current flowing through each link and the voltages across each of the resistors. To obtain the required equations, we need to first propose the directions of each link, select the ground node (node 0), and label the remaining nodes (nodes 1 to n). Based on the choices of current flow and node labels, we can form the node-link incidence matrix [=]n × m, which is a matrix composed of only 0, 1, and −1. The ith row of  refers to the ith node, whereas the j th column refers to the j th link. Note that the links containing only current sources are not included during the formulation of incidence matrix. (Instead, these links are involved only during the implementation of Kirchhoff’s current laws.) We set ψij = 1 if the current is flowing into node i along the j th link, and ψij = −1 if the current is flowing out of node i along the j th link. For the network shown in Figure 1.2, the incidence matrix is given by ⎛ ⎞ +1 −1 0 −1 0 0  = ⎝ 0 +1 −1 0 +1 0 ⎠ 0 0 0 +1 −1 −1 Let p i be the potential of node i with respect to ground and let e j be the potential difference along link j between nodes k and , that is, where ψkj = 0 and ψj = 0. Because the current flows from high to low potential, e = −T p If the j th link contains a voltage source s j , we assign a positive value if the polarity is from positive to negative along the chosen direction of the current flow. Let v j be the voltage across the j th resistor, then e=v+s Ohm’s law states that the voltage across the j th resistor is given by v j = i j Rj , where i j and Rj are the current and resistance in the j th link. In matrix form, we have ⎞ ⎛ 0 R1 ⎟ ⎜ .. v = Ri where R=⎝ ⎠ . 0

Rm

1.3 Properties of Matrix Operations

23

Let the current sources flowing out of the ith node be given by Aij , whereas those flowing into the ith node are given by Ai . Then the net current inflow at node i due only to current sources will be   Ai − Aij bi = 

j

Kirchhoff’s current law states that the net flow of current at the ith node is zero. Thus we have i + b = 0 In summary, for a given set of resistance, voltage sources, and current sources, we have enough information to find the potentials at each node, the voltage across each resistor, and the current flows along the links based on the chosen ground point and proposed current flows. To solve for the node potentials, we have e

=

v+s

−T p

=

Ri + s

−R  p − R s

=

i

−R−1 T p − R−1 s  −1 T  R  p

=

p

=

i = −b   b − R−1 s   −1 T −1  b − R−1 s R 

−1

T

−1

=

(1.20)

Using the values of p, we could find the voltages across the resistors, v = −T p − s

(1.21)

And finally, for the current flows, i = R−1 v

(1.22)

For the network shown in Figure 1.2, suppose the values for the resistors, voltage source, and current source are given by: {R1 , R2 , R3 , R4 , R5 , R6 } = {1 , 2 , 3 , 0.5 , 0.8 , 10 }, S1 = 1 v and A3,0 = 0.2 amp. Then the solution using equations (1.20) to (1.22) yields: ⎞ ⎛ ⎞ ⎛ 0.3882 0.3882 ⎜ 0.0932 ⎟ ⎜ 0.1864 ⎟ ⎛ ⎞ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ 0.6118 ⎟ ⎜ ⎟ ⎜ 0.1418 0.4254 ⎜ ⎟ ⎟ ⎜ ⎟ p = ⎝ 0.4254 ⎠ volts, v = ⎜ ⎜ 0.1475 ⎟ volts, and i = ⎜ 0.2950 ⎟ amps ⎟ ⎜ ⎟ ⎜ 0.4643 ⎟ ⎜ ⎟ ⎜ ⎝ 0.0486 ⎠ ⎝ 0.0389 ⎠ 0.0464 0.4643 Remarks: 1. R−1 is just a diagonal matrix containing the reciprocals of the diagonal elements  R.  −1 T of 2. R  is an n × n symmetric matrix, and its inverse is needed in equation (1.20). If n is large, it is often more efficient to approach the same

24

Matrix Algebra

problem using the numerical techniques that are covered in the next chapter, such as the conjugate gradient method. The last group of properties given in Table 1.3 involves the relationship between vectorization, matrix products, and Kronecker products. These properties are very useful in reformulating matrix equations in which the unknown matrices X do not exclusively appear on the right or left position of the products in the equation. For example, a form known as Sylvester matrix equation, which often results from control theory as well as in finite difference solutions, is given by QX + XR = C

(1.23)

where Q[=]N × N, R[=]M × M are C[=]N × M are constant matrices, whereas X[=]N × M is the unknown matrix. After inserting appropriate identity matrices, the properties can be used to obtain the following result:   = vec (C) vec QXIM + IN XR     vec QXIM + vec IN XR =     T IM ⊗ Q vec (X) + RT ⊗ IN vec (X) =   T IM ⊗ Q + RT ⊗ IN vec (X) = vec (C) T By setting A = IM ⊗ Q + RT ⊗ IN , x = vec (X), and b = vec (C), the problem can be recast as Ax = b.

In example 1.3, the finite difference equation resulted in the matrix equation given by

EXAMPLE 1.7.

T (k + 1) = AT (k) + T (k)B + C where A[=]N × N, B[=]M × M, C[=]N × M, and T [=]N × M. At equilibrium, T (k + 1) = T (k) = T eq , a constant matrix. Thus the matrix equation becomes T eq = AT eq + T eq B + C Using the vectorization properties in Table 1.4, we obtain           vec T eq = I[M] ⊗ A vec T eq + BT ⊗ I[N] vec T eq + vec (C) or Kx = b where

−→

x = K−1 b

K

=

x

=

    I[NM] − I[M] ⊗ A − BT ⊗ I[N]   vec T eq

b

=

vec (C)

After solving for x, T eq can be recovered by using the reshape operator, that is, T eq = reshape (x, N, M).

1.3 Properties of Matrix Operations

25

Table 1.6. Properties of determinants 1

Determinant of Products

      det AB = det A det B

2

Determinant of Triangular Matrices

   det A = N i=1 aii

3

Determinant of Transpose

    det AT = det A

4

Determinant of Inverses

  −1  det A−1 = det A

5†

Let B contain permuted columns of A based on sequence K

6†

Scaled ⎛ Columns: β1 a11 ⎜ .. B=⎜ ⎝ . ··· β1 aN1

⎞ βN a1N ⎟ .. ⎟ ⎠ . βN aNN

    det B = (K)det A where (K) is the permutation sign function       N det B = j =1 β j det A

⎛

a11 ⎜ . ⎜ . ⎝ . aN1

···

x1 + y1 .. . xN + yn

···

⎞ a1N .. ⎟ ⎟ . ⎠ aNN

⎛

7†

Multilinearity

8†

Linearly Dependent Columns

⎞ a11 x1 a1N ⎜ . .. .. ⎟ ⎟ =⎜ ⎝ .. ··· . ··· . ⎠ aN1 xN aNN ⎛ ⎞ a11 y1 a1N ⎜ . .. .. ⎟ ⎟ +⎜ ⎝ .. ··· . ··· . ⎠ aN1 yn aNN   det A = 0 N if for some γk = 0, j =1 γi A•,j = 0

† Using item 3 (i.e., that the transpose operation does not alter the determinant), a dual set of properties exists for items 5 to 8, in which the columns are replaced by rows.

1.3.2 Properties of Determinants Because the determinant is a very important matrix operation, we devote a separate table for the properties of determinants. A summary of the properties of determinants is given in Table 1.6. The proofs for these properties are given in Section A.4.2 as an appendix. Note that even though A and B may not commute, the determinants of both AB and BA are the same, that is,             det AB = det A det B = det B det A = det BA Several properties of determinants help to improve computational efficiency. For instance, the fact that the determinant of a triangular or diagonal matrix is just

26

Matrix Algebra

the product of the diagonal means that there is tremendous advantage to finding multiplicative factors that could “diagonalize” or “triagularize” the original matrix. Later, in Chapter 3, we try to find such a nonsingular T whose effect would be to make C = T −1 AT diagonal or triangular. Yet C and A will have the same determinant, that is,     det T −1 AT = det T −1 det (A) det (T ) =

1 det (A) det (T ) = det (A) det (T )

The last property in the list is one of the key application of determinants in linear algebra. It states that if the columns of a matrix are linearly dependent (defined next), then the determinant is zero. Definition 1.7. Vectors {v1 , v2 , . . . , vN } are linearly dependent if N 

γi vi = 0

(1.24)

j =1

for some γk = 0 This means that if {v1 , . . . , vN } is a linearly dependent set of vectors, then any of the vectors in the set can be represented as a linear combination of the other (N − 1) vectors. For instance, let ⎛ ⎛ ⎛ ⎞ ⎞ ⎞ 1 1 0 v1 = ⎝ 1 ⎠ v2 = ⎝ 2 ⎠ v3 = ⎝ 1 ⎠ 1 1 0   We can compute the determinant of V = v1 v2 v3 = 0 and conclude immediately that the columns are dependent. In fact, we check easily that v1 = v2 − v3 , v2 = v1 + v3 , or v3 = v2 − v1 .

Let a tetrahedron be described by four vertices in 3D space given by p1 , p2 , p3 , and p4 , as shown in Figure 1.3. Let v1 = p2 − p1 , v2 = p3 − p1 , and v3 = p4 − p1 form a 3 × 3 matrix  

EXAMPLE 1.8.

V =

v1

v2

v3

It can be shown using the techniques given in Section 4.1, together with Section 4.2, that the volume of the tetrahedron can be found by the determinant formula: ⎞ ⎛   1 ⎟ ⎜ Volume = abs ⎝det V ⎠ 6 For instance, let ⎛

⎞ 1 p1 = ⎝ 0 ⎠ 0

⎛

⎞ 1 p2 = ⎝ 1 ⎠ 0

⎛

⎞ 1 p3 = ⎝ 1 ⎠ 1

⎛

⎞ 0 p4 = ⎝ 1 ⎠ 1

1.3 Properties of Matrix Operations

27 p3

z

Figure 1.3. A tetrahedron described by four points: p1 , p2 , p3 , and p4 .

y

p

4

p2 x

p1

then the tetrahedron formed by vertices p1 , p2 , p3 , and p4 yields ⎛ ⎞ 0 0 −1 1 V =⎝ 1 1 1 ⎠ → Volume = 6 0 1 1  T p4 = 1 0 1 , If instead of p4 , we have by  ⎛ ⎞ 0 0 0 V = ⎝ 1 1 0 ⎠ → Volume = 0 0 1 1 which means p1 , p2 , p3 , and  p4 are coplanar, with v1 = v2 − v3 .

1.3.3 Matrix Inverse Formulas In this section, we include some of the formulas for the inverses of matrices and two important results: the matrix inversion lemma (known as the Woodbury formula) and Cramer’s rule. We start with the inverse of a diagonal matrix. The inverse of a diagonal matrix D is another diagonal matrix containing the reciprocals of the corresponding diagonal elements di , that is, ⎞−1 ⎛ −1 ⎞ ⎛ 0 0 d1 d1 ⎟ ⎟ ⎜ ⎜ .. .. D−1 = ⎝ (1.25) ⎠ =⎝ ⎠ . . 0

dN

0

−1 dN

Direct calculations can be used to show that DD−1 = D−1 D = I. Next, we have a formula for the inverse of a triangular matrix T of size N. For a triangular matrix T [=]N × N, let D be a diagonal matrix such that dii = tii and matrix K = D − T . Then,   N−1    KD−1 T −1 = D−1 I + (1.26)

LEMMA 1.3.

=1

PROOF.

Multiply (1.26) by T = D − K and expand,   N−1    N −1 −1 −1 TT = (D − K) D KD = I − KD−1 I+ =1 −1

but KD is a strictly triangular matrix that is nilpotent matrix of degree (N − 1) N  (see exercise E1.9), that is, KD−1 = 0.

28

Matrix Algebra

Next, we discuss an important result in matrix theory known as the matrix inversion lemma, also known as the Woodbury matrix formula. LEMMA 1.4.

PROOF.

Let A, C, and M = C−1 + DA−1 B be nonsingular, then  −1 (A + BCD)−1 = A−1 − A−1 B C−1 + DA−1 B DA−1

(1.27)

With M = C−1 + DA−1 B, let Q be the right hand side of (1.27), that is, Q = A−1 − A−1 BM−1 DA−1

(1.28)

Then, (A + BCD) Q

=

=

(AQ) + (BCDQ)  −1  AA − AA−1 BM−1 DA−1   + BCDA−1 − BCDA−1 BM−1 DA−1   I + BCDA−1 − B I + CDA−1 B M−1 DA−1   I + BCDA−1 − B CC−1 + CDA−1 B M−1 DA−1   I + BCDA−1 − BC C−1 + DA−1 B M−1 DA−1

=

I + BCDA−1 − BCMM−1 DA−1

=

I + BCDA−1 − BCDA−1

=

I

=

= =

In a similar fashion, one can also show that Q(A + BCD) = I.

Remark: The matrix inversion lemma given by (1.27) is usually applied in cases in which the inverse of A is already known and the size of C is significantly smaller than A.

EXAMPLE 1.9.

where

⎛

Let

1 T =⎝ 2 −1

⎛

⎞ 1 0 2 G=⎝ 2 2 0 ⎠ = T + wvT −1 −1 3 ⎞ 0 0 2 0 ⎠ −1 3

⎛

;

⎞ 1 w=⎝ 0 ⎠ 0

and

vT =



0

0

2



this means we split G into a triangular matrix T and a product of a column vector w and row vector vT . We can use lemma 1.3 to find ⎛ ⎞ 1 0 0 T −1 = ⎝ −1 1/2 0 ⎠ 0 1/6 1/3

1.3 Properties of Matrix Operations

29

Then with (1.27), −1  −1  −1 T −1 G−1 = T + w [1] vT = T −1 − T −1 w 1 + vT T −1 w v T ⎛ ⎞ 1 −1/3 −2/3 = ⎝ −1 5/6 2/3 ⎠ 0 1/6 1/3   where we took advantage of the fact that 1 + vT T −1 w [=]1 × 1. We complete this subsection with the discussion of a technique used in solving a subset of Ax = b. Suppose we want to solve for only one of the unknowns, for example, the kth element of x, for a given linear equation Ax = b. One could extract the kth element of x = A−1 b, but this involves the evaluation of A−1 , which can be computationally expensive. As it turns out, finding the inverse is unnecessary if only one unknown is needed, by using Cramer’s rule, as given by the following lemma: LEMMA 1.5.

Let A[=]N × N be nonsingular, then   det A[k,b] xk = det (A)

(1.29)

where A[k,b] is obtained A by replacing the kth column with b. PROOF.

Using (1.16), x = A−1 b can then be written as ⎛ ⎛ ⎞ ⎞ ⎛ b1 x1 cof(a11 ) · · · cof(aN1 ) ⎜ b2 1 ⎜ .. ⎟ ⎟⎜ ⎜ .. .. .. ⎝ . ⎠= ⎠ ⎜ .. ⎝ . . . ⎝ . det(A) xN cof(a1N ) · · · cof(aNN ) bN

or for the kth element,

n xk =

j =1

⎞ ⎟ ⎟ ⎟ ⎠

bj cof(akj )

det(A)

The numerator is just the determinant of a matrix, A[k,b] , which is obtained from A, with the kth column replaced by b.

EXAMPLE 1.10.

Let ⎛

1 A=⎝ 2 −1

⎞ 0 2 2 0 ⎠ −1 3

⎛

and

⎞ 2 b=⎝ 3 ⎠ 2

Then for Ax = b, the value of x2 can be found immediately using Cramer’s rule, ⎛ ⎞ 1 2 2 ⎝ 2 3 0 ⎠   −1 2 3 det A[2,b] 11 x2 = =⎛ ⎞= det (A) 6 1 0 2 ⎝ 2 ⎠ 2 0 −1 −1 3

30

Matrix Algebra

1.4 Block Matrix Operations A set of operations called block matrix operations (also known as partitioned matrix operations) takes advantage of special submatrix structures. The block operations are given as follows: 



A

B

C

D



det  det



E

F

G

H

A

0

C

D

A

B

C

D

A

B

C

D



 =



AE + BG

AF + BH

CE + DG

CF + DH

 (1.30)

=

    det A det D

=

  det(A) det D − CA−1 B

=

    det D det A − BD−1 C ; if D−1 exists





−1 =

(1.31) ; if A−1 exists

(1.32) (1.33)



W

X

Y

Z

(1.34)

where, W, X, Y , and Z depend on the two possible cases: Case 1: A and  = D − CA−1 B are nonsingular, then Z

=

−1

Y

=

−−1 CA−1 = −ZCA−1

X

=

−A−1 B−1 = −A−1 BZ

W

=

A−1 (I + B−1 CA−1 ) = A−1 (I − BY ) = (I − XC) A−1

(1.35)

Case 2: D and   = A − BD−1 C are nonsingular, then W

=

 −1

X

=

− −1 BD−1 = −WBD−1

Y

=

−D−1 C −1 = −D−1 CW

Z

=

D−1 (I + C −1 BD−1 ) = D−1 (I − CX) = (I − YB) D−1

(1.36)

The proofs of (1.30) through (1.36) are given in Section A.4.5. The matrices  = D − CA−1 B and   = A − BD−1 C are known as the Schur complements of A and D, respectively. EXAMPLE 1.11. Consider the open-loop process structure consisting of R process units as shown in Figure 1.4. The local state vector for process unit i is given T  by xi = x1i , · · · , xNi . For instance, xki could stand for the kth species in process unit i. The interaction among the process units is given by

A1 x1 + B1 x2

=

p1

Ci−1 xi−1 + Ai xi + Bi xi+1

=

pi

CR−1 xR−1 + AR xR

=

pR

if 1 < i < R

1.5 Matrix Calculus

31

Figure 1.4. An open-loop system of R process units.

where Ai , Bi , Ci [=]N × N and pi [=]N × 1. A block matrix description is Fx = p where

⎛

A1

⎜ ⎜ C1 F=⎜ ⎜ ⎝

B1 .. . .. .

0

⎞

0 ..

.

..

.

CR−1

BR−1 AR

⎛

⎞ x1 ⎜ ⎟ x = ⎝ ... ⎠ xR

⎟ ⎟ ⎟ ⎟ ⎠

⎛

⎞ p1 ⎜ ⎟ p = ⎝ ... ⎠ pR

As a numerical illustration, let N = 2 and R = 2 and ⎛ ⎛ ⎞ ⎞ −1 2 0 1 0.4 ⎜ 1 −2 ⎜0.6⎟ ⎟ 1 2 ⎟ and p = ⎜ ⎟ F=⎜ ⎝ 0 ⎝0.4⎠ 1 −2 1 ⎠ 1

1

0

−1

0.5

Using (1.36) to find F−1 , we have     −1 −0.5 −0.5 −0.0952 −1 −1 −1  D =  = A − BD C = 0 −1.0 0.3333 from which W

=

Y

=

 

−0.0952 0.3333 0.2857 0.2381

0.2857 0  0.1429 0.2857



 X

=

Z

=



and    W x1 −1 =F p= x= x2 Y

X Z





0.2857 0

−0.0952 0.3333

0.2857 0

−0.4286 0.1429

0.1429 −0.0476

 

⎛

⎞ ⎛ ⎞ 0.4 0.5333 ⎜ 0.5 ⎟ ⎜ 0.3333 ⎟ ⎜ ⎟ ⎜ ⎟ ⎝ 0.4 ⎠ = ⎝ 0.1 ⎠ 0.6

0.2667

Remark: Suppose we are interested in x1 (or x2 ); then one needs only the values of W and X (or Y and Z, respectively).

1.5 Matrix Calculus In this section, we establish the conventions and notations to be used in this book for derivatives and integrals of systems of multivariable functions and equations. For simplicity, we assume that the functions are sufficiently differentiable. The main

32

Matrix Algebra

advantage of matrix calculus is also to allow for compact notation and thus improve the tractability of calculating large systems of differential equations. This means that matrix algebra and matrix analysis tools can be used to study the solution and behavior of systems of differential equations, the numerical solution of systems of nonlinear algebraic equations, and the numerical optimization of multivariable functions.

1.5.1 Matrix of Univariable Functions Let A(t) be a matrix of univariable functions; then the derivative of A(t) with respect to t is defined as ⎛ ⎞ d(a11 ) d(a1M ) · · · ⎟   ⎜ dt dt ⎜ ⎟ d 1 . . . ⎜ ⎟ (1.37) .. .. .. A(t) = lim A(t + t) − A(t) = ⎜ ⎟

t→0 t dt ⎝ d(a ) d(aNM ) ⎠ N1 ··· dt dt Based on (1.37), we can obtain the various properties given in Table 1.7. For the derivative of determinants, the proof is given in Section A.4.6. Likewise, the integral of matrices of univariable functions is defined as follows:  tf ⎞ ⎛  tf a (t)dt · · · a (t)dt 11 1N ⎟ ⎜ t  tf t0 T −1 ⎟ ⎜ 0  ⎟ ⎜ . . .. .. .. A(t) dt = lim A (k t) t = ⎜ ⎟ (1.38) . ⎟ ⎜

t→0  tf t0 k=0 ⎠ ⎝  tf aN1 (t)dt · · · aNN (t)dt t0

t0

where T = (t f − t0 ) / t. Based on the linearity property of the integrals, we have the properties shown in Table 1.8.

Define the following function as the matrix exponential,   1 1 1 exp A(t) = I + A(t) + A(t)2 + A(t)2 + A(t)3 + · · · (1.39) 2! 2! 3!   then the derivative of exp A(t) is given by         d d 1 d d exp A(t) = 0 + A(t) + A(t) A(t) + A(t) A(t) + · · · dt dt 2! dt dt EXAMPLE 1.12.

In general, A(t) and its derivative are not commutative. However, for the special case in which A and its derivative commute, the matrix exponential simplifies to       d d d d 1 exp A(t) = A(t) + A(t) A(t) + A(t)2 A(t) + · · · dt dt dt 2! dt        d d A(t) = A(t) exp A(t) = exp A(t) dt dt One such case is when A(t) is diagonal. Another case is when A(t) = α(t)M, where M is a constant square matrix.

1.5 Matrix Calculus

33

Table 1.7. Properties of derivatives of matrices of univariable functions

1

Sum of Matrices

 d  d  d  M(t) + N(t) = M + N dt dt dt

2

Scalar Products

   dα d  d α(t)M(t) = M+α M dt dt dt

3

Matrix Products

   d  d  d M(t)N(t) = M N+M N dt dt dt

4

Hadamard Products

   d  d  d M(t) ◦ N(t) = M ◦N+M◦ N dt dt dt

5

Kronecker Products

   d  d  d M(t) ⊗ N(t) = M ⊗N+M⊗ N dt dt dt 



⎛ ⎜ ⎜ =⎜ ⎝

d A dt d C dt

6

Partitioned Matrices

d dt

7

Matrix Transpose

  d T d  A(t)T = A dt dt

8

Matrix Inverse

   d  d A(t)−1 = −A−1 A A−1 dt dt

A(t)

B(t)

C(t)

D(t)

d B dt d D dt

⎞ ⎟ ⎟ ⎟ ⎠

N       d  det A(t) = det  A k (t) dt k=1

where

9

Determinants

⎛

a11

⎜ ⎜ ⎜ ⎜ ⎜ dak1  A k = ⎜ ⎜ dt ⎜ ⎜ ⎝ an1

··· .. .

a1N

···

dakN dt

.. . ···

⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ← kth row ⎟ ⎟ ⎟ ⎠

aNN

Three vertices of a tetrahedron are stationary, namely p1 , p2 , and p3 . The last vertex p4 (t) moves as a function of t. As described in Example 1.8, the volume of the tetrahedron (after applying a transpose operation) is given by ⎛ ⎞  T ⎜ p2 − p1 ⎟ ⎜ ⎟ ⎜ ⎟ ⎜   T ⎟ 1 ⎜ ⎟ Vol = det ⎜ p3 − p1 ⎟ ⎜ ⎟ 6 ⎜ ⎟ ⎜  T ⎟ ⎝ ⎠ p4 − p1

EXAMPLE 1.13.

34

Matrix Algebra Table 1.8. Properties of integrals of matrices of univariable functions   1

Sum of Matrices

  2

Scalar Products

  3

Matrix Products

  4

Hadamard Products

  5

Kronecker Products

   M(t) + N(t) dt = Mdt + Ndt

 αM dt =



MN dt =

Partitioned Matrices

7

Matrix Transpose

if M is constant

⎧  ⎪ ⎪ ⎪ ⎪ ⎨M Ndt

if M is constant

  ⎪ ⎪ ⎪ ⎪ ⎩ Mdt N

if N is constant

⎧  ⎪ ⎪ ⎪ M ◦ Ndt ⎪ ⎨

if M is constant

 ⎪  ⎪ ⎪ ⎪ ⎩ Mdt ◦ N

if N is constant

 M(t) ⊗ N(t) dt =

A(t)

A(t)T

⎧  ⎪ ⎪ ⎪ M ⊗ Ndt ⎪ ⎨

if M is constant

 ⎪  ⎪ ⎪ ⎪ ⎩ Mdt ⊗ N

if N is constant

⎛  Adt ⎜  ⎜ B(t) ⎜ dt = ⎜  ⎜ D(t) ⎝ Cdt

C(t)

 

if α is constant

  ⎪ ⎪ ⎪ ⎪ ⎪ αdt M ⎩

 M(t) ◦ N(t) dt =

  6

⎧  ⎪ ⎪ ⎪ α Mdt ⎪ ⎪ ⎨



⎞

Bdt ⎟ ⎟ ⎟ ⎟  ⎟ Ddt ⎠

T

 dt =



A dt

Using the formula for the derivative of determinants (cf. property 9 in Table 1.7), the rate of change of Vol per change in t is given by ⎞ ⎛ ⎛ ⎞ ⎟ ⎜  T ⎟ ⎜ ⎜ ⎜ p2 − p1 ⎟ ⎟ ⎜ ⎜ ⎟ ⎟ ⎜ ⎜ ⎟ ⎟ ⎜ ⎜  ⎟ ⎟ ⎜  T ⎟ ⎟ ⎜ 1 ⎜ d ⎜ ⎟ ⎟ Vol = ⎜ 0 + 0 + det ⎜ p3 − p1 ⎟ ⎟ dt 6 ⎜ ⎜ ⎟ ⎟ ⎜ ⎜ ⎟ ⎟ ⎜ ⎜ ⎟ ⎟   ⎜ d ⎝ ⎠ ⎟ T ⎟ ⎜ p 4 ⎟ ⎜ dt ⎠ ⎝

1.5 Matrix Calculus

For instance, let the points be given by ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ 1 0.5 1 p1 = ⎝2⎠ p2 = ⎝ 1 ⎠ p3 = ⎝1⎠ 0 0 0 then

⎛

−0.5 d 1 ⎝ Vol = 0 + 0 + det 0 dt 6 2

35

⎛

⎞ 2t + 3 p4 = ⎝−t + 1⎠ t+5

−1 −1 −1

⎞ 0 1 0 ⎠= 12 1

Let f (θ) = pT Q(θ)p, where p is constant and Q(θ) is a square matrix. Then the integral of f (θ) can be evaluated as  π  π f (θ)dθ = pT Q(θ)p dθ

EXAMPLE 1.14.

0

0

=



T

p



π

Q(θ) dθ p 0



For a specific example,    π   p1 cos(θ) sin(θ) p 1 p 2@ dθ = p 1 p − sin(θ) cos(θ) 2 0

@p 2

    0 −2 p1 =0 p2 2 0

1.5.2 Derivatives of Multivariable Functions Let xi , i = 1, . . . , N, be independent variables collected in a column vector as ⎛ ⎞ x1 ⎜ .. ⎟ x=⎝ . ⎠ xN then a multivariable, scalar function f of these variables is denoted by     f x = f x1 , x2 , . . . , xM whereas a vector of multivariable scalar functions f(x) is also arranged in a column vector as ⎞ ⎛ ⎞ ⎛ f 1 (x1 , x2 , . . . , xM ) f 1 (x) ⎟ ⎜ ⎟ ⎜ .. f (x) = ⎝ ... ⎠ = ⎝ ⎠ . f N (x)

f N (x1 , x2 , . . . , xM )

We denote a row vector of length M known as the gradient vector, which is the partial derivatives of f (x) by   ∂f d ∂f , ... , f (x) = (1.40) dx ∂x1 ∂xM

36

Matrix Algebra

When applied to each function in a vector of N functions, we obtain an N × M matrix known as the Jacobian matrix, ⎛

⎞ ⎛ ∂f 1 f (x) ⎜ 1 ⎟ ⎜ ∂x 1 ⎟ d d ⎜ ⎜ .. ⎟ ⎜ .. ⎜ f (x) = = ⎜ ⎟ . dx dx ⎜ . ⎟ ⎜ ⎝ ⎠ ⎝ ∂f m f M (x) ∂x1

··· ..

.

···

∂f 1 ∂xn .. .

∂f m ∂xn

⎞ ⎟ ⎟ ⎟ ⎟ ⎠

(1.41)

A set of first-order consecutive chemical reactions A → B → C occur in an isothermal continuously stirred tank reactor under constant volume operation. The dynamic equations describing the component balance of each compound are given by   dxA V = F in xA,in − xA − kAVxA dt   dxB V = F in xB,in − xB − kBVxB + kAVxA dt   dxC V = F in xC,in − xC − kCVxC + kBVxB dt

EXAMPLE 1.15.

where xA, xB, and xC are the concentrations of A, B, and C, respectively. The other variables in the model are treated as constants. We can collect these concentrations in a column vector x as ⎛ ⎞ ⎛ ⎞ x1 xA x = ⎝xB ⎠ = ⎝x2 ⎠ xC x3 then the differential equations can be represented in matrix form as   d   x =f x dt where ⎛  ⎞ ⎛ ⎞  F in  f1 x xA,in − x1 − kAx1 ⎟ ⎜ ⎜ ⎟ ⎟   ⎜  ⎟ ⎜ FV   ⎟ ⎜ in ⎜ ⎟ f x = ⎜ f2 x ⎟ = ⎜ xB,in − x2 − kBx2 + kAx1 ⎟ ⎟ ⎜ V ⎜ ⎟ ⎝  ⎠ ⎝ F  ⎠  in f3 x xC,in − x3 − kCx3 + kBx2 V which can be further recast in matrix product form as:   f x = Kx + b where K and b are constant matrices given by ⎛ ⎞ −kA 0 0 K = ⎝ kA ; −kB 0 ⎠ −kC 0 kB

⎞ ⎛ x F in ⎝ A,in ⎠ b= xB,in V xC,in

1.5 Matrix Calculus

37

The term Kx, which is just a collection of N linear combination of the elements of x, is said to have the linear form. Furthermore, note that K can also be obtained   by taking the Jacobian matrix of f x , that is, ⎛

∂ f1 ⎜ ⎜ ∂x1 ⎜ ⎜ ∂ f2 d   ⎜ f x =⎜ ⎜ dx ⎜ ∂x1 ⎜ ⎜ ∂f 3 ⎝ ∂x1

∂ f1 ∂x2 ∂ f2 ∂x2 ∂ f3 ∂x2

⎞ ⎛ ∂ f1 ⎟ ⎜ − kA ∂x3 ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎜ ∂ f2 ⎟ ⎟ = ⎜ kA ⎟ ∂x3 ⎟ ⎜ ⎜ ⎟ ⎜ ⎟ ∂ f3 ⎠ ⎜ ⎝ 0 ∂x3

⎞ 0

0

−kB

0

kB

−kC

⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠

We show later that when f (x) is expanded through a Taylor series, there will always be a term that has the linear form, precisely due to the Jacobian matrix being evaluated at a prescribed point. We define the operator d2 /dx2 on f (x) as ⎛ ⎞ ⎛ ⎞ ∂f/∂x1   T 2  d ⎜ d d d ⎜ . ⎟ ⎟ = f (x) f (x) ⎠ = ⎝ ⎝ .. ⎠ 2 dx dx dx dx ∂f/∂xN ⎛ ⎞ ∂2 f ∂2 f ··· ⎜ ∂x 2 ∂x1 ∂xN ⎟ 1 ⎜ ⎟ ⎜ ⎟ .. .. .. = ⎜ ⎟ . . . ⎜ ⎟ 2 ⎝ ∂2 f ⎠ ∂ f ··· ∂xN ∂x1 ∂xN 2

(1.42)

This matrix of second-order derivatives is known as the Hessian matrix of f (x). ∂2 f ∂2 f Note that because = , Hessian matrices are symmetric. ∂xi ∂x j ∂x j ∂xi Both the gradients and the Hessians are useful in evaluating local maxima of multivariable functions. Specifically, we have the following theorem that gives the sufficient condition for a local minimum or local maximum. Let f (x) be a multivariable function that is twice differentiable in x. A point x∗ yields a local minimum value for f (x) if   d f (x) = 0T (1.43) dx ∗ x=x

THEOREM 1.1.

and

⎛

⎞   ⎜ d2 ⎟  ⎟ vT ⎜ ⎝ dx2 f (x) ∗ ⎠ v > 0 x=x

for all v = 0

Likewise, A point x∗ yields a local maximum value for f (x) if   d = 0T f (x) dx x=x∗

(1.44)

(1.45)

38

Matrix Algebra

and

⎛

⎞  ⎜ d2 ⎟   ⎟ vT ⎜ ⎝ dx2 f (x) ∗ ⎠ v < 0 x=x

PROOF.

for all v = 0

(1.46)

(See A.4.9)

The conditions given in (1.44) and (1.46) are also known as positive definiteness condition and negative definiteness conditions, respectively. Definition 1.8. An N × N matrix A is positive definite, denoted (A > 0), if x∗ Ax > 0

for all x = 0

(1.47)

and A is positive semi-definite if (1.48) x∗ Ax ≥ 0 for all x For negative definite matrices, we need to just note from (1.46) that a matrix M is negative definite if −M is positive definite. We include a brief discussion on some tests for positive definiteness in Section A.5 as an appendix. EXAMPLE 1.16.

Consider the function   2 2 f x1 , x2 = e−(x1 +1) e−(x2 −1)

A plot of f (x1 , x2 ) is shown in Figure 1.5. The gradient is given by     d ∂f ∂f −(x1 +1)2 −(x2 −1)2 f (x) = = −2e e (x1 + 1) (x2 − 1) dx ∂x1 ∂x2 The gradient is zero at the point (x1 , x2 ) = (−1, 1), which makes that point a candidate local maximum or minimum. Furthermore, the Hessian of f (x) is given by ⎞ ⎛  2  ⎜ 4x1 + 8x1 + 2 4 (x1 + 1) (x2 − 1) ⎟ ⎟ ⎜ d2 f ⎟ ⎜ = ⎟ f (x1 , x2 ) ⎜ ⎜ dx2  2  ⎟ ⎠ ⎝ 4 (x1 + 1) (x2 − 1) 4x − 8x2 + 2 2

At (x1 , x2 ) = (−1, 1), the Hessian becomes   d2 f −2 0 = 0 −2 dx2 and for v = 0, 

v1

v2





−2 0

0 −2

    v1 = −2 v21 + v22 < 0 v2

which satisfies the negative definiteness condition. Thus the point (x1 , x2 ) = (−1, 1) is a local maximum. This can be seen to be the case from Figure 1.5.

1.6 Sparse Matrices

39

f(x1,x2) 1

Figure 1.5. The surface plot for 2 2 e−(x1 +1) e−(x2 −1) .

f (x1 , x2 ) =

0.5

0 3 2 x2

1 0 -1 -3

-2

-1

For the special cases of linear and quadratic forms, the derivatives, gradients, and Hessians have special formulas as given by the following lemma: LEMMA 1.6.

PROOF.

Let A[=]M × N and x[=]N × 1, then   d Ax = A dx   d For A[=]N × N, xT Ax = dx   d2 T x Ax = dx2

(1.49) xT

  A + AT

A + AT

(1.50) (1.51)

(See section A.4.7 )

Remark: Equations (1.49) to (1.51) are used in the next chapter during the solution to the least squares problem.

1.6 Sparse Matrices We end this chapter with a brief discussion on sparse matrices because several real-world problems involve large matrices that contain a very high percentage of zero entries. Computer programs such as MATLAB include special commands to handle the calculation of sparse matrices. For instance, consider the finite difference approximation of d2 u/dx2 , d2 u ui+1 − 2ui + ui−1 ≈ dx2

x2 This leads to a matrix representation Au, where ⎛ −2 1 ⎜ .. . 1 ⎜ ⎜ 1 −2 A= ⎜ 2 . .

x ⎝ .. .. 0 1

0

⎞

⎟ ⎟ ⎟ ⎟ 1⎠ −2

0 x1

1

40

Matrix Algebra Table 1.9. Some MATLAB commands for sparse matrix operation MATLAB Command

Description Creates a sparse matrix of size N×M with rowA, colA, and nzA as the vectors for row index, column index, and nonzero elements, respectively Converts a full formatted matrix S to sparse matrix format Returns the row and column indices of nonzero elements of matrix A Visualization of the sparsity pattern of matrix A Creates a sparse formatted identity matrix of size N Creates an M×N sparse matrix by placing columns of V along the diagonals specified by d Performs sparse matrix operations and leaves the result in sparse matrix format Evaluates the functions of all elements (zero and nonzero) but leaves the results in sparse matrix format

A=sparse(rowA,colA,nzA,N,M)

A=sparse(S) [rowA,colA]=find(A) spy(A) speye(N) A=spdiags(V,d,M,N)

Operations: +, -, *,\,ˆ Functions: exp,sin,cos, tan,log,kron,inv,

The fraction of nonzero elements is (3N − 2)/N2 . For instance, with N = 100, this fraction is, approximately 1%. Two features of sparse matrices can immediately lead to computational advantages. The first feature is that the storage of only nonzero elements of sparse matrices will result in very significant savings by avoiding the storage of zero elements. The second feature is that when performing matrix-vector products, it should not be necessary to multiply the zero elements of the matrix with that of the vector. For instance, ⎛

β = vT w =



⎜ ⎜ ⎜ 2 1 0 0 0 3 ⎜ ⎜ ⎜ ⎝

a b c d e f

⎞ ⎟ ⎟ ⎟  ⎟= 2 ⎟ ⎟ ⎠

1

3



⎛

⎞ a ⎝ b ⎠ f

We discuss only the coordinate format because it is the most flexible and the simplest approach, although other schemes can have further significant storage savings. In this format, the nonzero elements can be collected in a vector along with two vectors of the same length: one vector for the row indices and the other vector for the column indices. For example, let matrix A be given as ⎛

0 ⎜ 0 A=⎜ ⎝ d 0

a 0 0 0

0 0 0 c 0 e 0 g

⎞ b 0 ⎟ ⎟ f ⎠ 0

1.7 Exercises

41

then the three vectors nzA, rowA, and colA, indicating nonzero elements, row indices, and column indices, respectively, are given by ⎛ ⎜ ⎜ ⎜ ⎜ nzA = ⎜ ⎜ ⎜ ⎜ ⎝

a b c d e f g

⎞

⎛

⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠

⎜ ⎜ ⎜ ⎜ rowA = ⎜ ⎜ ⎜ ⎜ ⎝

1 1 2 3 3 3 4

⎞

⎛

⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠

⎜ ⎜ ⎜ ⎜ colA = ⎜ ⎜ ⎜ ⎜ ⎝

2 5 4 1 4 5 4

⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠

This implies that the storage requirements will be thrice the length of nzA. As a guideline, for the storage under the coordinate formats to be worthwhile, the fraction of nonzero elements must be less than 31 . Some MATLAB commands for sparse matrices are listed on Table 1.9. In the next chapter, we often note the solution methods for the linear equation Ax = b that have been developed to preserve as much of the sparsity of matrices, because the amount of computations can be significantly reduced if sparsity of A is present. 1.7 EXERCISES

E1.1. Show that AT A, AAT , and A + AT are symmetric, and A − AT is skew symmetric. E1.2. Let A[=]10 × 50, B[=]50 × 20, and α a scalar. From a computational view, is it better to compute (αA) B, A (αB), or α (AB)? Explain your conclusions, including any possible exceptions to your conclusion. E1.3. Prove or disprove the following properties for matrix traces: tr (A + B) = tr (A) + tr (B) tr (αA) = αtr (A) for scalar α. tr AT = tr (A). tr (ABC) = tr (CAB) = tr (BCA), assuming A[=]N × M, B[=]M × P, C[=]P × N. If, in addition, A, B, and C are symmetric, then tr (ABC) = tr (ACB). 5. If A is symmetric and B is skew symmetric, then tr (AB) = 0. 6. tr (A ⊗ B) = tr (A) tr (B).

1. 2. 3. 4.

E1.4. Let A[=]N × M and B[=]M × N; show that     det I[N] − AB = det I[M] − BA (Hint: use (1.32) and (1.33)) E1.5. For a nonsingular matrix G, define the relative gain array of G as R = G ◦ G−T where G−T = (G−1 )T . Prove the following properties for R: 1. The sum of all elements along any row or along any column is 1, that is, n  i=1

rij

=

1

for all i

42

Matrix Algebra n 

rij

=

1

for all j

j =1

2. if G is triangular, then R = I. Note: The relative gain array is used in process control theory to determine the best pairing between manipulated and controlled variables that would reduce the interaction among the control loops. E1.6. If A2 = A then A is said to be idempotent. 1. Show that if A is idempotent, then I − A and Ak , k > 1, are also idempotent.     2. Show that if A is idempotent, then either det A = 1 or det A = 0. Furthermore, show that the only nonsingular idempotent matrices are the identity matrices. 3. Projection matrices are idempotent matrices that are also Hermitian. Verify that the following matrix is a projection matrix: ⎛ ⎞ 2 −2 −2 1⎝ A= −2 5 −1 ⎠ 6 −2 −1 5 and evaluate its determinant. E1.7. Let p, s, and t be three points in the (x, y)-plane, that is,       s t px s= x t= x p= py sy ty A circumcircle for p, s, and t is a circle that would pass through through all three points, as shown in Figure 1.6.

Figure 1.6. Circumcircle that passes through p, s, and t.

Let ρ{p,s,t} and c{p,s,t} = circumcircle.



cx

cy

T

be the radius and center point for the

1. Show that the radius of the circumcircle has to satisfy ρ2{p,s,t} = (p − c)T (p − c) = (t − c)T (t − c) = (s − c)T (s − c)

(1.52)

2. Show that (1.52) can be combined to yield the following matrix equation:     (p − s)T 1 pT p − sT s c= 2 s T s − tT t (s − t)T 3. Using the preceding results, find the radius and center of the circumcircle for       1 1 0 p= s= t= 0 0 1 Plot the points and the circumcircle to verify your solution.

1.7 Exercises

43

4. Find the circumcircle for   3 t= 5

  2 s= 2

 1 p= −1 

What do you conclude about this example ? Note: Circumcircles are used in mesh-triangulation methods for finite elements solution of partial differential equations. One specific method is called the Delaunay triangulation method. For more details, see Chapter 14. E1.8. Using Q defined in (1.28), complete the proof of the matrix inversion lemma given in (1.27) by showing that indeed Q(A + BCD) = I. E1.9. If Ar = 0, with Ar−1 = 0 for some positive integer r, then A is said to be nilpotent of index r. A strictly triangular matrix is a triangular matrix whose diagonal elements are all zero. Show that strictly triangular matrices are nilpotent. E1.10. Matrix A is said to be orthogonal if AT A = AAT = I. 1. Show that permutation matrices, P, are orthogonal where ⎛

⎞ eTk1 ⎜ . ⎟ . ⎟ P=⎜ ⎝ . ⎠

;

k1 = · · · = kN

eTkN and ⎞ 0 ⎜ .. ⎟ ⎜ . ⎟ ⎟ ⎜ ⎜ 0 ⎟ ⎟ ⎜ th ⎟ ei = ⎜ ⎜ 1 ⎟ ← i element ⎜ 0 ⎟ ⎟ ⎜ ⎜ . ⎟ . ⎝ . ⎠ ⎛

0 2. Show that the products of orthogonal matrices are also orthogonal. E1.11. Matrix A is said to be unitary if A∗ A = AA∗ = I. Show that the normalized Fourier matrix, given by ⎛

f 11 1 ⎜ . F = √ ⎝ .. N f N1

··· .. . ···

⎞ f 1N .. ⎟ ; . ⎠ f NN

f k = e−2πi(k−1)(−1)/N

is unitary. E1.12. Determine whether the following statement is true or false, and explain: “Let Q be skew-symmetric, then xT Qx = 0 for all vectors x.”

44

Matrix Algebra

E1.13. A matrix that is useful for the method of Gaussian elimination is the matrix obtained by replacing the kth column of an identity matrix by vector w as follows: ⎞ ⎛ 1 w1 0 .. ⎟ ⎜ .. ⎟ ⎜ . . ⎟ ⎜ ⎟ ⎜ 1 w k−1 ⎟ ⎜ ⎟  EL(k, w) = ⎜ w k ⎟ ⎜ ⎟ ⎜ w 1 k+1 ⎟ ⎜ ⎟ ⎜ . . .. .. ⎠ ⎝ 0 wN 1 1. Evaluate the determinant of  EL(k, w). EL(k, w) is given by 2. Assuming wk = 0, show that the inverse of  ⎞ ⎛ 1 −(w1 /wk ) 0 .. ⎟ ⎜ .. ⎟ ⎜ . . ⎟ ⎜ ⎟ ⎜ /w ) 1 −(w k−1 k ⎟ ⎜ −1 ⎟ ⎜  EL(k, w) = ⎜ (1/wk ) ⎟ ⎟ ⎜ −(wk+1 /wk ) 1 ⎟ ⎜ ⎟ ⎜ .. .. ⎠ ⎝ . . −(wN /wk )

0

1

3. Consider a matrix A[=]N × M with a nonzero element in the kth row and th column, that is, ak = 0. Based on ak , define the elements of vector g(ak ) as follows: ⎧ ai ⎪ − if i = k ⎪ ⎪ ⎨ ak gi = ⎪ 1 ⎪ ⎪ if i = k ⎩ ak   Discuss the effects of premultiplying A by  EL k, g(ak ) . (Hint: Use the following specific example, ⎛ ⎞ 4 −1 3 2 A=⎝ 2 1 5 2 ⎠ 1 0 2 1     Construct  EL 3, g(3, 1) and  EL 1, g(1, 4) , then obtain the products  ELA and observe the effects. Using these results, infer as well as prove the general results.) E1.14. The Vandermonde matrix is a matrix having a special structure given by ⎛ n−1 ⎞ λ1 λn−2 · · · λ1 1 1 ⎜ n−1 ⎟ ⎜ λ2 λn−2 · · · λ2 1 ⎟ 2 ⎜ ⎟ V =⎜ . .. .. .. ⎟ .. . ⎝ . . . . . ⎠ n−2 λ · · · λ 1 λn−1 n n n and a determinant given by    det V = (1.53) (λi − λ j ) i 1

(1.66)

x0 contains the mole fractions in the distillate stream, p contains the reciprocal relative volatilities 1/αi , and R is the (scalar) reflux ratio. 1. Evaluate the composition of the liquid out of the third stage, x3 , using the following values: ⎛ ⎞ ⎛ ⎞ 0.2 7 p = ⎝ 4 ⎠ , x0 = ⎝ 0.1 ⎠ , R = 0.6 0.7 1 9

This problem is adapted from the example given in N. Amundson, Mathematical Methods in Chemical Engineering, Volume 1, Matrices and Their Applications, Prentice Hall, Englewood Cliffs, NJ, 1966, pp. 149–157.

50

Matrix Algebra

2. Show that for k > 1, the iterative formula (1.66) can be recast as qk = Aqk−1 where A = D + pxT0 and

⎛ ⎜ D=⎝

(1.67) ⎞

0

Rp 1 ..

⎟ ⎠

.

0

Rp n

Thus show that qk = Ak−1 p 3. For matrix A defined in (1.67), show that the determinant and the adjugate of (A − λI), where λ is a scalar variable, are given by det (A − λI)

=

adj (A − λI)

=

ψλ ωλ   λ ψλ I − pxT0 λ ωλ

where ωλ =

n 



(Rp i − λ)

;

i=1

⎛

⎜ λ = ⎝

(Rp 1 − λ)−1 0

n  x0i p i ψλ = 1 + Rp i − λ i=1 ⎞ 0 ⎟ .. ⎠ . −1 (Rp n − λ)

(1.68) (1.69) 

(Hint: Use (1.63) for the determinant and (1.27) for the adjugate formula.)10 E1.23. Consider the tridiagonal matrix given by ⎛ a1 b1 ⎜ .. ⎜c1 . . . . TN = ⎜ ⎜ .. .. ⎝ . . 0 cN−1

0

⎞

⎟ ⎟ ⎟ ⎟ bN−1 ⎠ aN

(1.70)

Define T 0 = [1] and T 1 = [a1 ]. 1. Show that the determinant of T k , for k > 1 is given by the following recursive formula (also known as the continuant equation):       (1.71) det T k = ak det T k−1 − bk−1 ck−1 det T k−2 2. Verify this formula by finding the determinant of ⎛ ⎞ 1 −1 0 A = ⎝ −2 2 1 ⎠ 0 2 −2 10

This formula for the adjugate can be used further to obtain a closed formula for xn based on Sylvester’s formula.

1.7 Exercises

51

3. For the special case that a1 = a2 = · · · = ak =a, b1 = b2 = · · · = bk = b and c1 = c2 = · · · = ck = c. Let Dk = det T k , then (1.71) becomes a difference equation given by Dk+2 − aDk+1 + bcDk = 0 (1.72) √ subject to D0 = 1 and D1 = a. Let = a2 − 4bc. Show that the solution of (1.72) is given by ⎧ k+1 ⎪ − (a − )k+1 ⎪ ⎪ (a + ) if = 0 ⎪ ⎨ 2k+1

Dk = (1.73) ⎪  a k ⎪ ⎪ ⎪ if = 0 ⎩ (1 + k) 2 (Hint: The solution given in (1.73) can be derived from (1.72) by treating it as a difference equation subject to the two initial conditions. These methods are given in Section 7.4. For this problem, all that is needed is to check whether (1.73) will satisfy (1.72).) E1.24. The general equation of an ellipse is given by  2  2 x y xy + −2 cos δ = sin2 δ a1 a2 a1 a2

(1.74)

where a1 , a2 , sin(δ) = 0. Let v = (x, y)T , find matrix A such that equation (1.74) can be written as vT Av = 1 (Note: vT Av = 1 is the general equation of a conic, and if A is positive definite, then the conic is an ellipse.) E1.25. Show that tr (AB) = tr (BA) (assuming conformability conditions are met). E1.26. Let f (x1 , x2 ) be given by       f (x) = exp −3 (x1 − 1)2 + (x2 + 1)2 − exp −3 (x1 + 1)2 + (x2 − 1)2 Find the gradient df/dx at x = (0, 0)T , x = (1, −1)T and x = (−1, 1)T . Also, find the Hessian at x = (1, −1)T and x = (−1, 1)T and determine whether they are positive or negative definite. E1.27. Determine which of the following matrices are positive definite:       3 4 3 4 0 2 A= B= C= 1 2 1 0 −2 0 E1.28. Let A be a square matrix containing a zero in the main diagonal. Can A be positive definite? Why or why not? E1.29. Prove the following equality

    d d d A B+A B (AB) = dt dt dt E1.30. If A is nonsingular, prove that d −1 dA −1 A = −A−1 A dt dt E1.31. A Marquardt vector update with a scalar parameter μ is defined by  −1 T p (μ) = − J T J + μI J F

(1.75)

52

Matrix Algebra

where J [=]m × n and F[=]m × 1 are constant matrices with m ≥ n. This vector is useful in solving unconstrained minimization algorithms. ! Another scalar function φ (μ) is used to indicate the proximity of p = pT p to a fixed value δ, " φ (μ) = pT p − δ Show that

 −1 pT J T J + μI p dφ ! =− T dμ p p E1.32. Let P be a simply connected polygon (i.e., containing no holes) in a 2D plane described by points (xi , yi ) indexed in a counterclockwise manner. Then the area of polygon P, areaP , can be obtained by the following calculations:11 ⎞ ⎛       1 x x x2 x3 x1 ⎟ x ⎜ areaP = abs ⎝det 1 + det 2 + · · · + det N ⎠ y1 y2 y2 y3 yN y1 2 (1.76) 1. Show that (1.76) can be recast as follows:     1 areaP = abs xT Sh − ShT y 2 where Sh is the shift matrix given by ⎛ ⎞ 0 1 0 ⎜ .. ⎟ .. ⎜ ⎟ . Sh = ⎜ . ⎟ ⎝ 0 0 1 ⎠ 1

0

···

(1.77)

0

2. Verify (1.77) for the simple case of the triangle determined by (x, y)1 = (1, 1), (x, y)2 = (2, 1), and (x, y)3 = (1.8, 2). Furthermore, notice that if we replace the third point by (x, y)3 = (a, 2) for arbitrary real values for a, we will obtain the same area. Explain why this is the case, both for the figure and the formula. 3. The points covering the area of crystal growth was obtained from a scanned image and given in Table 1.10. Plot the polygon and then obtain the area using (1.77). E1.33. Consider the generalized Poisson equation in a rectangular domain L × W given by ∂2u ∂2u + 2 = (x, y)u + (x, y) ∂x2 ∂y subject to the boundary conditions, u(0, y) = f 0 (y) u(L, y) = f L(y)

u(x, 0) = g 0 (x) u(x, W) = g W (x)

Using the central difference approximation, ∂2u un+1,m − 2un,m + un−1,m ∂2u un,m+1 − 2un,m + un,m−1 ≈ and ≈ ∂x2

x2 ∂y2

y2 11

M. G. Stone, A mnemonic for areas of polygons, The American Mathematical Monthly, vol. 93, no. 6, (1986), pp. 479–480.

1.7 Exercises

53

Table 1.10. Envelope points for crystal growth

x

y

x

y

x

y

0.2293 0.2984 0.3583 0.3813 0.3813 0.3975 0.4459 0.4988 0.5311 0.5749 0.6555

0.3991 0.4488 0.4781 0.5541 0.6155 0.6915 0.7325 0.6827 0.5892 0.5687 0.5482

0.6970 0.7431 0.7385 0.6901 0.6210 0.5634 0.5173 0.5081 0.4965 0.4666

0.5307 0.5015 0.4693 0.4459 0.3991 0.4079 0.3670 0.2675 0.1944 0.1535

0.4343 0.4067 0.3813 0.3629 0.3445 0.3353 0.3168 0.2800 0.2569 0.2339

0.1447 0.1681 0.2208 0.2675 0.2939 0.3114 0.3260 0.3319 0.3406 0.3582

where n = 1, 2, · · · , N, m = 1, 2, · · · , M, x = L/(N + 1), y = W/(M + 1) and un,m = u(n x, m y). Show that the finite difference approximation will yield the following matrix equation: AU + UB = Q ◦ U + H

(1.78)

where A = (1/ x )T [N×N] , B = (1/ y )T [M×M] , Q = [qnm ], ⎞ ⎛ −2 1 0 ⎟ ⎜ ⎟ ⎜ 1 −2 . . . ⎟ [=]K × K T [K×K] = ⎜ ⎟ ⎜ .. .. ⎝ . . 1⎠ 0 1 −2 ⎛ ⎞ f 0,1 · · · f 0,M ⎛ ⎞ ⎜ 0 γ11 · · · γ1M ··· 0 ⎟ ⎜ ⎟ 1 ⎜ ⎜ ⎟ .. ⎟ − .. .. H = ⎝ ... ⎜ ⎟ ⎠ . . . ⎟

x2 ⎜ ⎝ γN1 · · · γNM 0 ··· 0 ⎠ f L,1 · · · f L,M ⎛ ⎞ 0 g W,1 g 0,1 0 1 ⎜ ⎟ .. . .. − 2 ⎝ ... ⎠ . · · · .. .

y 0 g W,N g 0,N 0 2

2

with qnm = (n x, m y), γnm = (n x, m y), f 0,m = f 0 (m y), f L,m = f L(m y), g 0,n = g 0 (n x), and g W,n = g W (n x). Furthermore, show that (1.78) can be transformed to a linear equation given by Z vec (U) = vec (H)   where Z = IN ⊗ A + B ⊗ IM − diag vec (Q) . E1.34. Prove that all the alternative quadratic forms given in example 1.4 are equivalent, that is, xT Qx = xT Lx = xT Rx.

2

Solution of Multiple Equations

One of the most basic applications of matrices is the solution of multiple equations. Generally, problems involving multiple equations can be categorized as either linear or nonlinear types. If the problems involve only linear equations, then they can be readily formulated as Ax = b, and different matrix approaches can be used to find the vector of unknowns given by x. When the problem is nonlinear, more complex approaches are needed. Numerical approaches to the solution of nonlinear equations, such as the Newton method and its variants, also take advantage of matrix equations. In this chapter, we first discuss the solution of the linear equation Ax = b. This includes direct and indirect methods. The indirect methods are also known as iterative methods. The distinguishing feature between these two types of approaches is that direct methods (or noniterative) methods obtain the solution using various techniques such as reduction by elimination, factorization, forward or backward substitution, matrix splitting, or direct inversion. Conversely, the indirect (iterative) methods require an initial guess for the solution, and the solution is improved using iterative algorithms until the solution meets some specified criterion of maximum number of iterations or minimum tolerance on the errors. The most direct approach is to simply apply the inverse formula given by x = A−1 b This is a good approach as long as the inverse can be found easily, for example, when the matrix is orthogonal or unitary. Also, if the matrix is diagonal or triangular, Section 1.3.3 gives some direct formulas for their inverse. However, in general, the computation of the matrix inverse using the adjoint formula given in (1.16) is not the method of choice, specially for large systems. In Section 2.1, we discuss the first direct method known as the Gauss-Jordan elimination method. This simple procedure focuses on finding matrices Q and W such that QAW will yield a block matrix with an identity matrix in the upper left corner and zero everywhere else. In Section 2.2, a similar approach known as the LU decomposition method is discussed. Here, matrix A is factored as A = LU, with L and U being upper triangular and lower triangular matrices, respectively. The triangular structures of L and U allow for quick computation of the unknowns via forward and backward substitutions. Other methods such as matrix splitting techniques, which 54

2.1 Gauss-Jordan Elimination

55

take advantage of special structures of A, are also discussed, with details given the appendix as Section B.6. In Section 2.4, we switch to indirect (iterative) methods, which include: Jacobi, Gauss-Seidel, and the succesive over-relaxation (SOR). Other iterative methods, such as conjugate-gradient (CG) and generalized minimal residual (GMRES) methods, are also discussed briefly, with details given in the appendices as Sections B.11 and B.12.2. In Section 2.5, we obtain the least-squares solution. These are useful in parameter estimation of models based on data. We also include a method for handling leastsquares problems that involve linear equality constraints. Having explored the various methods for solving linear equations, we turn our attention to the solution of multiple nonlinear equations in Section 2.9. We limit our discussion to numerical solutions based on Newton’s methods, because they involve the application of matrix equations.

2.1 Gauss-Jordan Elimination Assuming A is nonsingular, the Gauss-Jordan Elimination method basically involves the determination of two nonsingular matrices Q and W such that QAW = I. Assuming both matrices have been found, the linear equation can be solved as follows: Ax

=

b

QAx

=

Qb

−1

x

=

Qb

W −1 x

=

Qb

x

=

WQb

(QAW) W

(2.1)

This method can also be used to find the inverse, A−1 = WQ

(2.2)

If A is nonsingular, there are several values of Q and W that will satisfy QAW = I.1

EXAMPLE 2.1.

Consider the following: ⎛ ⎞⎛ ⎞ ⎛ ⎞ 2 −1 1 x1 −1 ⎝ 4 5 −1 ⎠ ⎝ x2 ⎠ = ⎝ 15 ⎠ x3 −1 3 2 3

Leaving the details for obtaining Q and W for now, note that the following matrices will satisfy QAW = I: ⎛ ⎞ ⎛ ⎞ 0 1/5 0 0 1 13/17 Q=⎝ 0 3/17 −5/17 ⎠ W = ⎝ 1 −4/5 −7/17 ⎠ 17/50 −1/10 7/25 0 0 1 1

 = TQ and W  W  = WT −1 for any nonsingular T will also yield QA  =I Suppose QAW = I, then Q   Q. and A−1 = WQ = W

56

Solution of Multiple Equations

then (2.1) yields the solution

⎛

⎞ ⎞ ⎛ −1 1 x = WQ ⎝ 15 ⎠ = ⎝ 2 ⎠ 3 −1

Even when A is singular or A is a non-square N × M matrix, the same approach can be generalized to include these cases. That is, there exist nonsingular matrices Q and W such that QAW yields a partitioned matrix consisting of an r × r identity matrix on the upper left portion and zero matrices on the remaining partitions, ⎛ ⎞ ⎜ Ir ⎜ ⎜ QAW = ⎜ ⎜ ⎝ 0 [N−r,r]

0[r,M−r] 0[N−r,M−r]

⎟ ⎟ ⎟ ⎟ ⎟ ⎠

(2.3)

When A is nonsingular, r = N = M; otherwise, r ≤ min(N, M). The value of r, which is the size of the identity in the upper left corner, is known as the rank of matrix A. The rank indicates how many columns or rows of A are linearly independent.

2.1.1 Evaluation of Q and W The classical approach of the Gauss-Jordan elimination method is based on finding a group of elementary column and row matrix operations (cf. Section 1.2.2.1) that would sequentially replace the kth row and kth column with zeros except for the diagonal entry, which is replaced by 1 (hence the name “elimination”). For instance, let A be given by ⎛ ⎞ 1 1 1 A = ⎝ −1 2 3 ⎠ 2 4 3 With

⎛

0 ⎝ EL = 0 1 we have

0 1 0

⎞ 1/4 −1/2 ⎠ −1/4

⎛

and

⎛

1 ⎝ ELAER = 0 0

0 ⎝ ER = 1 0

0 −2 1/2

1 −1/2 0

⎞ 0 −3/4 ⎠ 1

⎞ 0 3/2 ⎠ 1/4

Matrix EL and ER can be easily formulated using the formulas given in (B.3) to (B.6). The next stage is to extract the lower right block matrix and then apply the elimination process once more. The process stops when the lower right blocks are all zeros. The complete details of the classic Gauss-Jordan elimination are given in Section B.1 as an appendix. In addition, a MATLAB program gauss_jordan.m is available on the book’s webpage.

2.1 Gauss-Jordan Elimination

57

Another alternative to finding matrices Q and W is based on the Singular Value Decomposition (SVD) method. This approach has been found to be more stable than the Gauss-Jordan elimination approach. Details on how to use SVD for this purpose are given in Section B.2 as an appendix.

2.1.2 Singular A and Partial Rank Cases When r < min(N, M), two outcomes are possible: either there are an infinite number of solutions or there are no solutions. Starting with Ax = b, using Q and W, that satisfy (2.3),   (QAW) W −1 x ⎞⎛ ⎞

⎛ ⎜ Ir ⎜ ⎜ ⎜ ⎜ ⎝ 0 [N−r,r]

0[r,M−r] 0[N−r,M−r]

⎟ ⎜ yupper ⎟⎜ ⎟⎜ ⎟⎜ ⎟⎜ ⎠⎝ y lower

⎟ ⎟ ⎟ ⎟ ⎟ ⎠

=

=

Qb ⎛

⎞

⎜ Qupper ⎜ ⎜ ⎜ ⎜ ⎝ Q lower

⎟ ⎟ ⎟ ⎟b ⎟ ⎠

(2.4)

where y = W −1 x. The first r equations are then given by yupper = Qupper b

(2.5)

whereas the last (N − r) equations of (2.4) are given by 0 yupper + 0 ylower = Qlower b

(2.6)

This means that if (Qlower b) = 0, no exact solution exists. However, if (Qlower b) = 0, ylower can be any arbitrary vector, implying an infinite set of solutions. Equivalently, the existence of solutions can be determined by checking whether the rank r of A is the same as the rank of the enlarged matrix formed by appending A with b, that is, solutions exist if and only if  rank



A

b

 = rank

 A

(2.7)

Note that when b = 0, both (2.6) and (2.7) are trivially satisfied. Suppose Qlower b = 0 in (2.6). Using yupper from (2.5) and appending it with (M − r) arbitrary constants ci for ylower , we have ⎛ ⎜ Qupper b ⎜ ⎜ ⎜ ⎜ −1 W x=y=⎜ ⎜ c1 ⎜ .. ⎜ ⎝ . cM−r

⎛

⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠

=⇒

⎜ Qupper b ⎜ ⎜ ⎜ ⎜ x=W⎜ ⎜ c1 ⎜ .. ⎜ ⎝ . cM−r

⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠

58

Solution of Multiple Equations

Let WL and WR be matrices formed by the first r columns and the last (M − r) columns of W, respectively; then the solutions are given by ⎛ ⎞  x=

WL

EXAMPLE 2.2.

WR

⎜ Qupper b ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ c1 ⎜ .. ⎜ ⎝ . cM−r

⎟ ⎟ ⎛ ⎞ ⎟ c1 ⎟ ⎟ ⎜ . ⎟ ⎟ = WL Qupper b + WR ⎝ .. ⎠ ⎟ ⎟ cN−r ⎟ ⎠

Consider the following equation for Ax = b, ⎛ ⎞⎛ ⎞ ⎛ ⎞ 1 2 1 x1 0 ⎝ 3 2 4 ⎠ ⎝ x2 ⎠ = ⎝ 1 ⎠ x3 2 4 2 0

The values of r, Q, and W can be found to be r = 2 and ⎛ ⎞ ⎛ 0 0 1/4 0 0 Q = ⎝ 0 1/3 −1/6 ⎠ W = ⎝ 1 −1/2 1 0 −1/2 0 1 and

(2.8)

⎛

0 WL = ⎝ 1 0

⎞ 0 −1/2 ⎠ 1

⎞ 1 −1/6 ⎠ −2/3

⎛

and

⎞ 1 WR = ⎝ −1/6 ⎠ −2/3

Because Qlower b = 0, we have infinite solutions. Using (2.8), ⎞ ⎞ ⎛ ⎛ 1 0 ⎟ ⎟ ⎜ ⎜ x = ⎝ −1/6 ⎠ + ⎝ −1/6 ⎠ c1 1/3

−2/3

Remarks: 1. The difference DOF = (M − r) is also known as the degree of freedom, and it determines the number of arbitrary constants available (assuming that solution is possible). 2. Full-rank non-square matrices can be further classified as either full-columnrank (when r = M < N), also known as full-tall matrices, or full-row-rank (when r = N < M), also known as full-wide matrices. For the full-tall case, assuming that Qlower b = 0, (2.8) implies that only one solution is possible, because DOF = 0. However, for the full wide-case, the condition given in (2.6) is not necessary to check because Qlower is no longer available. This means that for full-wide matrices, infinite solutions are guaranteed with DOF = M − r. 3. In Section 2.5, the linear parameter estimation problem involves full-tall matrices A, in which the rows are associated with data points whereas the columns are associated with the number of unknown parameters x1 , . . . , xN . It is very likely

2.2 LU Decomposition

59

that the Ax = b will not satisfy (2.6), that is, there will not be an exact solution. The problem will have to be relaxed and modified to the search for x that would minimize the difference (b − Ax) based on the Euclidean norm. The modified problem is called the least-squares problem, which is discussed later in Section 2.5.

2.2 LU Decomposition Instead of using the matrices Q and W resulting from the Gauss-Jordan elimination method, one can use a factorization of a square matrix A known as the LU decomposition in which L and U are lower and upper triangular matrices, respectively, such that A = LU

(2.9)

The special structures of L and U allow for a two-phase approach to solving the linear problem Ax = b: a forward-substitution phase followed by a backward-substitution phase. Let y = Ux, then the LU decomposition in (2.9) results in

⎛ ⎜ ⎜ ⎜ ⎝

11 21 .. .

22 .. .

N1

N2

0 ..

. ···

⎞⎛ ⎟⎜ ⎟⎜ ⎟⎜ ⎠⎝

NN

Ax

=

b

L (Ux)

=

b

Ly ⎞

=

b ⎛

y1 y2 .. .

⎟ ⎟ ⎟ ⎠

=

⎜ ⎜ ⎜ ⎝

yN

b1 b2 .. .

⎞ ⎟ ⎟ ⎟ ⎠

bN

Assuming L is nonsingular, the forward-substitution phase is the sequential evaluation of y1 , y2 , . . . , yN ,  bN − N−1 b1 b2 − 21 y1 i=1 Ni yi y1 = → y2 = → · · · → yN = (2.10) 11 11 NN Once y has been found, the backward-substitution phase works similarly to sequentially obtain xN , xN−1 , . . . , x1 , that is, with ⎛ ⎜ ⎜ ⎜ ⎝

u11

··· .. .

0

u1,N−1 .. .

u1N .. .

uN−1,N−1

uN−1,N

⎞⎛

x1 .. .

⎟⎜ ⎟⎜ ⎟⎜ ⎠ ⎝ xN−1

uNN

Ux ⎞ ⎟ ⎟ ⎟ ⎠

=

=

y ⎛

y1 .. .

⎜ ⎜ ⎜ ⎝ yN−1

xN

→ xN−1

yN−1 − uN−1,N xN = uN−1,N−1

→ · · · → x1 =

⎟ ⎟ ⎟ ⎠

yN

then yN xN = uNN

⎞

y1 −

N i=2

u1i xi

u11 (2.11)

In some textbooks, LU decomposition is synonymous with Gaussian elimination.

60

Solution of Multiple Equations EXAMPLE 2.3.

Consider the linear equation ⎛ ⎞ ⎛ ⎞ 2 1 −1 −1 ⎝ 4 0 2 ⎠x = ⎝ 8 ⎠ 6 2 2 8

One LU decomposition by ⎛ 1 ⎝ L= 2 3

(using methods discussed in the next section) is given 0 2 1

⎞ 0 0 ⎠ 1

⎛

and

2 ⎝ U= 0 0

1 −1 0

⎞ −1 2 ⎠ 3

The forward substitution yields the solution of Ly = b, −1 8 − 2(−1) 8 − 3(−1) − 1(5) = −1 → y2 = = 5 → y3 = =6 1 2 1 and the backward-substitution gives the solution of Ux = y, y1 =

x3 =

6 =2 3

→

x2 =

5 − 2(2) = −1 −1

→

x1 =

−1 − (−1)(2) − 1(−1) =1 2

Remarks: In MATLAB, one can use the backslash ( \) operation for dealing with either forward or backward substitution.2 Thus, assuming we found lower and upper matrices L and U such that A = LU, then for solving the equation Ax = b, the forward substitution to find y = L−1 b can be implemented in MATLAB as: y=L\b;. This is then followed by x = U −1 y, the backward substitution that can be implemented in MATLAB as: x = U\y.

2.2.1 Crout, Doolittle, and Choleski Methods for LU Decomposition To find an LU decomposition of A, we can use the equations for aij resulting from the special structure of L and U,

aij =

N  k=1

ik ukj

⎧ i−1 ⎪ ii uij + k=1 ik ukj ⎪ ⎪ ⎪ ⎪ ⎨ i−1 = ii u jj + k=1 ik ukj ⎪ ⎪ ⎪ ⎪ ⎪  j −1 ⎩ ij u jj + k=1 ik ukj

if i < j if i = j

(2.12)

if i > j

Various LU decomposition algorithms are possible by imposing some additional conditions. Crout’s method uses the condition u jj = 1, whereas Doolittle’s method uses the condition ii = 1. Another LU decomposition known as Choleski’s method is possible if A is symmetric and positive definite, by setting L = U T , that is, LLT = A. These three methods are summarized in Table 2.1. 2

When the “backslash” operation is used in MATLAB, different cases are assessed first by MATLAB, that is, the function determines first whether it is sparse, banded, tridiagonal, triangular, full, partialranked, and so forth, and then it chooses the appropriate algorithms. More details are available from the help file, mldivide.

2.2 LU Decomposition

61

Table 2.1. Methods for LU decomposition Name

Crout’s Method

Doolittle’s Method

Choleski’s Method

Algorithm (For p = 1, . . . , N) u pp

=

ip

=

u pj

=

 pp

=

u pj

=

ip

=

 pp

=

ip

=

1  

1 

aip − a pj −

 p −1 k=1

 p −1 k=1

ik ukp

 

 pk ukj / pp

  pk ukj    p −1 aip − k=1 ik ukp /u pp a pj −

 p −1 k=1

"  p −1 a pp − k=1 2pk    p −1 aip − k=1 ik  pk / pp

for i = p, . . . , N for j = p + 1, . . . , N

for j = p, . . . , N for i = p + 1, . . . , N

for i = p + 1, . . . , N

Remarks: 1. The nonzero elements of L and U are evaluated column-wise and row-wise, respectively. For example, in Crout’s method, at the pth stage, we first set u pp = 1 and then evaluate  pp . Thereafter, the pth column of L and the pth row of U are filled in. 2. The advantage of Choleski’s method over the other two methods is that the required storage is reduced by half. However, Choleski’s method requires the square root operation. Other factorizations are available for symmetric positive definite matrices that avoid the square root operation, for example, A = LDLT , where D is a diagonal matrix. 3. Because the methods use the reciprocal of either ii or u jj , pivoting, that is, permutations of the rows and columns of A, are sometimes needed, as discussed next. For nonsingular matrices, pivoting is often needed unless A is positive definite or A is diagonally dominant, that is, if  |aii | ≥ (2.13) [aij ] j =i

A simple rule for pivoting at the pth stage is to maximize | pp | (for Crout’s method) or |u pp | (for Doolittle’s method) by permuting the last (N − p ) columns or rows. In MATLAB, the command for LU factorization is given by: [L,U,P]=lu(A), where LU = PA and P is a permutation matrix. (A MATLAB file crout_rowpiv.m is available on the book’s webpage that implements Crout’s method with row-wise pivoting.) If A is singular, permutation of both rows and columns of A are needed, that is, one must find permutation matrices PL and PR such that    L1 0 U1 C PLAPR = (2.14) B 0 0 I(N−r) where L1 and U 1 are lower and upper triangular matrices, respectively.

62

Solution of Multiple Equations

A major concern occurs when using LU factorization of large sparse matrices. During the factorization, the zero entries of A may be replaced by nonzero entries in L or U. This situation is known as “fill-ins” by L or U. This means that LU factorization could potentially lose the storage and efficiency advantages gained by the sparsity of A. For example, let A be given by ⎞ ⎛ 1 1 1 1 1 1 1 ⎜ 1 0 1 0 0 0 0 ⎟ ⎟ ⎜ ⎜ 1 1 0 0 0 0 0 ⎟ ⎟ ⎜ ⎟ A=⎜ ⎜ 1 0 0 1 0 0 0 ⎟ ⎜ 1 0 0 0 1 0 0 ⎟ ⎟ ⎜ ⎝ 1 0 0 0 0 1 0 ⎠ 1

0

0

0

0

0

then Crout’s method will yield the following factors: ⎛ ⎛ ⎞ 1 0 0 0 0 0 0 ⎜ ⎜ ⎟ ⎜ 1 −1 ⎜ 0 0 0 0 0 ⎟ ⎜ ⎜ ⎟ ⎜ 1 ⎜ ⎟ 0 −1 0 0 0 0 ⎟ ⎜ ⎜ ⎜ ⎜ ⎟ L = ⎜ 1 −1 −1 2 0 0 0 ⎟ and U = ⎜ ⎜ ⎜ ⎟ ⎜ 1 −1 −1 1 3 0 0 ⎟ ⎜ ⎜ ⎜ ⎟ 2 ⎜ ⎜ ⎟ 1 4 ⎝ 1 −1 −1 1 2 3 0 ⎠ ⎝ 1

−1

−1

1

1 2

1 3

5 4

1

1

1

1

1

1

1

0

1

0

1

1

1

0

0

1

1

1

1

1

⎞

0

0

0

1

1 2

0

0

0

0

1

1 2 1 3

0

0

0

0

0

1

⎟ 1 ⎟ ⎟ 1 ⎟ ⎟ 1 ⎟ ⎟ 2 ⎟ 1 ⎟ 3 ⎟ 1 ⎟ ⎠ 4

0

0

0

0

0

0

1

This example shows a significant number of fill-ins compared with the original A. Thus, if one wants to use LU factors to solve the linear equation Ax = b, where A is sparse, some additional preconditioning is often performed before solution. For instance, one could find a permutation matrix P that transforms the problem into   PAPT (Px) = (Pb) →  A x = b   such that  A = PAPT attains a new structure that would allow minimal fill-ins in L and U. One popular approach is to find  A that has a small bandwidth. Definition 2.1. For a square matrix A of size N, the left matrix bandwidth of A is the value      BWleft (A) = max k = j − i  i ≤ j, aij = 0 Likewise, the right matrix bandwidth of A is the value      BWright (A) = max k = i − j  i ≥ j, aij = 0 The maximum of left or right bandwidths   BW (A) = max BWleft (A) , BWright (A) is known as the matrix bandwidth of A. In short, the left bandwidth is the lowest level of the subdiagonal (diagonals below the main diagonal) that is nonzero, whereas the right bandwidth is the highest

2.2 LU Decomposition

level of super-diagonal (diagonals above the main instance, let ⎛ 1 0 0 0 ⎜ 0 2 0 0 ⎜ ⎜ 0 0 0 1 A=⎜ ⎜ 0 0 0 −1 ⎜ ⎝ 0 −1 0 0 0 0 0 0

63

diagonal) that is nonzero. For 0 0 0 0 3 0

0 0 0 0 0 1

⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠

  then BWleft (A) = 3 and BWright (A) = 1, so BW (A) = 3. Note also that BWleft AT =   1 and BWright AT = 3, so BW AT = 3. Thus, for symmetric matrices, the bandwidth, the left bandwidth, and the right bandwidth are all the same. From the LU factorization calculations used in either Crout’s or Doolittle’s algorithm, it can be shown that the L will have the same left bandwidth as A, and U will have the same right bandwidth of A. This means that the fill-ins by L and U factors can be controlled by reducing the bandwidth of A. One algorithm for obtaining the permutation P such that  A = PAPT has a smaller bandwidth is the Reverse Cuthill-McKee Reordering algorithm. Details of this algorithm are given in Section B.4 as an appendix. In MATLAB, the command is given by p=symrcm(A), where p contains the sequence of the desired permutation.

2.2.2 Thomas Algorithm Let A have a tri-diagonal structure given by ⎛ a1 b1 ⎜ .. ⎜ c1 . . . . A=⎜ ⎜ .. .. ⎝ . . 0 cN−1

0

bN−1 aN

⎞ ⎟ ⎟ ⎟ ⎟ ⎠

(2.15)

Assuming that no pivoting is needed, for example, when A is diagonally dominant, then the LU decomposition via Crout’s method will yield bi-diagonal matrices L and U given by ⎛ ⎛ ⎞ ⎞ 1 (b1 /z1 ) z1 0 0 ⎜ ⎜ ⎟ ⎟ .. .. ⎜ ⎜ c1 . . . ⎟ ⎟ . . ⎟ and U = ⎜ ⎟ L=⎜ ⎜ ⎜ ⎟ ⎟ .. .. .. ⎝ ⎝ ⎠ . bN−1 /zN−1 ⎠ . . 0 cN−1 zN 0 1 where z1 = a1

and

zk = ak −

bk−1 ck−1 zk−1

,

k = 2, . . . , N

(2.16)

The forward-substitution phase for solving Ly = v is then given by y1 =

v1 z1

and

yk =

1 (vk − ck−1 yk−1 ) zk

whereas the backward-substitution phase is given by bk xk+1 xN = yN and xk = yk − , zk

,

k = 2, . . . , N

k = N − 1, . . . , 1

(2.17)

(2.18)

64

Solution of Multiple Equations

One implication of (2.16), (2.17), and (2.18) is that there is no need to form matrices L and U explicitly. Furthermore, from (2.16), we note that the storage space used for ak can also be relieved for use by zk . The method just described is known as the Thomas algorithm and it is often used in solving linear equations resulting from onedimensional finite difference methods. (A MATLAB code thomas.m is available on the book’s webpage, which implements the Thomas algorithm and takes advantage of storage savings.)

EXAMPLE 2.4.

Consider the following two-point boundary value problem: h2 (x)

d2 u du + h1 (x) + h0 (x)u = f (x) 2 dx dx

subject to u(0) = u0 and u(L) = uL. We can use the finite difference approximations of the derivatives given by d2 u uk+1 − 2uk + uk−1 ≈ 2 dx

x2

and

du uk+1 − uk−1 ≈ dx 2 x

Let h2,k = h2 (k x), h1,k = h1 (k x), h0,k = h0 (k x), f k = f (k x) and uk = u(k x), where x = L/(N + 1). This results in the following linear equation: ⎞⎛ ⎛ ⎞ ⎛ ⎞ a1 b1 0 u1 w1 ⎟⎜ ⎜ ⎜ ⎟ .. .. ⎟ ⎜ u2 ⎟ ⎜ ⎟ ⎜ w2 ⎟ . . ⎟⎜ ⎜ c1 ⎜ ⎟ ⎟ ⎟⎜ ⎜ =⎜ . ⎟ ⎟ ⎜ .. ⎟ ⎜ ⎜ ⎟ ⎟ . . . ⎟ ⎜ .. .. bN−1 ⎠ ⎝ . ⎠ ⎝ . ⎠ ⎝ uN wN 0 cN−1 aN where, ak = h0,k −

2h2,k

x2

h2,k h1,k h2,k+1 h1,k+1 ; bk = + ; ck = − 2 2

x 2 x

x 2 x ⎧ ⎪ ⎪ f 1 − c0 u0 if k = 1 ⎪ ⎪ ⎪ ⎨ fk if 2 ≤ k ≤ N − 1 wk = ⎪ ⎪ ⎪ ⎪ ⎪ ⎩ f N − bN uL if k = N

For a specific example, let L = 4, h2 (x) = x2 , h1 (x) = x, h0 (x) = 7 and     f (x) = −9x2 + 3x − 7 e−3x + 5x3 − 15x2 + 40x e−x with boundary values u0 = −1 and uL = 0.3663. Using N = 100, we apply the Thomas algorithm and obtain zT

=

(5.000, −1.900, 8.737, . . . , −9.268, −9.434, −9.603)

yT

=

(−0.824, 0.032, −0.108, . . . , −0.0047, −0.0054, 0.3773)

T

=

(−1.000, −0.698, −0.423, . . . , 0.389, 0.377, 0.366)

u

The exact solution is u(x) = 5xe−x − e−3x . Plots of the numerical solution and the exact solution are shown in Figure 2.1.

2.3 Direct Matrix Splitting

65

2

1.5

1

u(x) 0.5

0 Exact Finite Difference

−0.5

−1 0

0.5

1

1.5

2

2.5

3

3.5

4

x Figure 2.1. A plot of the numerical solution using finite difference approximation and the exact solution for Example 2.4.

A block matrix LU decomposition is also possible. The details for this are given in Section B.5. Likewise, a block matrix version of the Thomas algorithm is available and is left as an exercise (cf. exercise E2.16).

2.3 Direct Matrix Splitting In some cases, a matrix A can be split into two matrices M and S, that is, A = M + S, such that M is easy to invert (e.g., diagonal, triangular, tri-diagonal, block diagonal, block triangular, or block tri-diagonal).3 One possibility is when the rows and columns of S = A − M can be permuted to contain large numbers of zero rows or zero columns; then an efficient method known as diakoptic method can be used to solve the linear equation. Consider the problem Ax = b, where ⎛ ⎞ −2 1 0 0 0 ⎜ 1 −2 1 0 0 ⎟ ⎜ ⎟ 1 −2 1 0 ⎟ A=⎜ ⎜ −2 ⎟ and ⎝ 3 0 1 −2 1 ⎠

EXAMPLE 2.5.

−2

0

0

1

3

0

0

1

−2

⎜ ⎜ b=⎜ ⎜ ⎝

−2

We can split matrix A as A = M + S with ⎛ ⎞ −2 1 0 0 0 ⎜ 1 −2 1 0 0 ⎟ ⎜ ⎟ ⎜ 1 −2 1 0 ⎟ M=⎜ 0 ⎟ and ⎝ 0 0 1 −2 1 ⎠ 0

⎛

⎛ ⎜ ⎜ S=⎜ ⎜ ⎝

0 0 −2 3 −2

0 0 0 0 0

1 3 0 1 2

0 0 0 0 0

⎞ ⎟ ⎟ ⎟ ⎟ ⎠

0 0 0 0 0

0 0 0 0 0

⎞ ⎟ ⎟ ⎟ ⎟ ⎠

Formulas for finding the inverse of triangular matrices are given in (1.26) and can be extended to block triangular matrices as in Exercise E1.18.

66

Solution of Multiple Equations

Then we have (M + S)x = b

(I + M−1 S)x = M−1 b

→

Since M is tridiagonal and only the first column of S contains nonzero terms, M−1 need to be applied only on the first column of S and on b, using the Thomas algorithm. This results in a lower triangular problem which can be solved using forward substitution, ⎞ ⎛ ⎞ ⎛ ⎞ ⎛ −2.625 −3.5 1.333 0 0 0 0 ⎜ −4.250 ⎟ ⎜ −6.0 ⎟ ⎜ 0.667 1 0 0 0 ⎟ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎜ 1.000 0 1 0 0 ⎟ x = ⎜ −5.5 ⎟ → x = ⎜ −2.875 ⎟ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎝ −6.750 ⎠ ⎝ 5.0 ⎠ ⎝ −0.667 0 0 1 0 ⎠ −1.750 −3.5 0.667 0 0 0 1

Another possibility is when A has a special “block-arrow” structure. These problems occur often when domain decomposition methods are applied during the finite difference solutions of partial differential equations. In those cases, the method of Schur complements can be used. Details for both general diakoptic methods and Schur complement methods are included in Section B.6 as an appendix.

2.4 Iterative Solution Methods In this section, we briefly discuss a group of methods known as stationary iterative methods. These methods are given by the form x(k+1) = Hx(k) + c

(2.19)

where H and c are constant, and x(k) is the kth iteration for the solution x. A solution is accepted when the iterations have converged to a vector x∗ , for example, when the norm given by # $ N   $ (k+1) (k) 2 (k+1) (k) x −x =% xi − xi

(2.20)

i=1

is less than some tolerance . Based on the solution of difference equation (cf. section 7.4), one can show that stationary iterative methods will converge if another index based on the matrix H, called the spectral radius, is less than one. The spectral radius of H, denoted by ρ (H), is defined as  N   ρ (H) = max λi (H) i=1

(2.21)

where λi is the ith eigenvalue of H. We discuss eigenvalues and their role in stability in more detail in Section 3.8, Example 3.10. For now, we simply state the the eigenvalues of a matrix H[=]N × N are the N roots of the polynomial equation, det (λI − H) = 0

(2.22)

2.4 Iterative Solution Methods

67

Examples of the stationary iterative methods include the Jacobi method, GaussSeidel method, and successive over-relaxation (SOR) methods. All three methods result from a matrix splitting A = A1 + A2

(2.23)

such that A1 x = v or A−1 1 is easy to solve. In Section 2.3, we showed how matrix splitting can be used for direct solution of linear equations if A possesses special structures such as those given in Sections B.6.1 and B.6.2. However, when the matri ces are sparse, yet lacking in special structures, the inverse of I + A−1 1 A2 may not be as easy to calculate directly and is likely to significantly increase the number of “fill-ins.” In this case, an iterative approach is a viable alternative.

2.4.1 Jacobi Method The Jacobi method chooses the split by extracting the diagonal of A, that is, ⎛ ⎜ A1 = ⎝

0

a11 .. 0

⎛

⎞

⎜ ⎜ A2 = ⎜ ⎝

⎟ ⎠

. aN,N

0 a21 .. .

a12 0 .. .

··· ··· .. .

a1N a2N .. .

aN1

aN2

···

0

⎞ ⎟ ⎟ ⎟ ⎠

The matrix equation can then be rearranged as follows: Ax = (A1 + A2 ) x

=

b

A1 x

=

b − A2 x

x

=

−1 A−1 1 b − A1 A2 x

Starting with an initial guess of x(0) , the Jacobi iteration is given by −1 (k−1) x(k) = A−1 1 b − A1 A2 x

(2.24)

where x(k) is the kth iterate. One often writes (2.24) in the indexed notation as  xki = vi − sij xk−1 (2.25) j j =i

where vi =

bi aii

sij =

aij aii

j = i

Some important issues need to be addressed: 1. Pivoting is needed to make sure that aii = 0 for all i. 2. Depending on A, the Jacobi iterations may still not converge. The first issue is straightforward, and usually pivoting is used to move the elements of maximum absolute values to the main diagonal. The second issue is more challenging. Because the Jacobi method is a stationary iterative method,

68

Solution of Multiple Equations

  convergence will depend on the spectral radius of A−1 1 A2 . If the spectral radius is not less than 1, additional techniques are needed.

2.4.2 Gauss-Seidel Method The Gauss-Seidel method can be viewed as a simple improvement over the Jacobi method. Recall (2.25) and note that the formula for the kth iteration of the ith element, (k) (k) xi , means that the elements, x j for j = 1, . . . , (i − 1), can already be used on the right-hand side of (2.25). Thus the Gauss-Seidel method is given by xki = vi −



sij xkj −

j 1 is needed to gain an improvement over Gauss-Seidel. The optimal choice among the values given was for ω = 1.3. However, the SOR method was divergent for ω = 1.7. Note that A is not a symmetric positive definite matrix. In Section 2.7 and Section 2.8, we show two more iterative methods: the conjugate gradient method and the generalized minimal residual (GMRES) method, respectively. These methods have good convergence properties and do not sacrifice the sparsity of the matrices. The iterations involved in the conjugate gradient method and GMRES are based on optimization problems that are framed under the linear-algebra perspective of the linear equation Ax = b. Thus, in the next section, we include a brief review of linear algebraic concepts, which also leads to the least-squares solution.

4

10

ω=1.7

2

10

ω=1.5

0

10

Err

ω=0.8

Figure 2.3. Convergence for the SOR method for ω = {0.8, 1, 1.3, 1.5, 1.7} applied to the equation in Example 2.5.

−2

10

ω=1.0

ω=1.3

−4

10

−6

10

0

10

20

30

k (Iterations)

40

50

2.5 Least-Squares Solution

71

2.5 Least-Squares Solution In this section, we are changing to the perspective of solving Ax = b using a linear algebra interpretation. That is, we are now looking for x as the weights under which the column vectors of A will be combined to match b as closely as possible. We have included a brief review of some basic terms and notation of linear vector algebra in Section B.7 as an appendix. For a fixed A and b, we define the residual vector, r, as r = b − Ax

(2.31)

The Euclidean norm of the residual vector will be used as a measure of closeness between b and Ax, where the norm of a vector v is defined by √ v = v∗ v (2.32) As noted in Section B.7, one can find x such that r = 0 only if b resides in the span of the columns of A. Otherwise, we have to settle for the search of  x that would minimize the norm of r. Recall from Theorem 1.1 that for  x to yield a local optimum value of f (x), it is sufficient to have have the gradient of f be zero and the Hessian of f be positive definite at x =  x. Furthermore, for the special case in which f (x) is given by f (x) = x∗ Qx + w∗ x + c

(2.33)

where Q is Hermitian and positive definite, the minimum will be global. To see this, first set the gradient to zero,  d  f = x∗ Q + w∗ = 0[1×M] → Q x = −w dx x=x Because Q is nonsingular, the value for  x will be unique. Let us now apply this result to minimize the norm of the residual vector. The value of  x that minimizes r will also minimize r2 . Thus, r2

=

r∗ r = (b − Ax)∗ (b − Ax)

=

(b∗ b) + x∗ A∗ Ax − 2b∗ Ax

where we used the fact that b∗ Ax is symmetric. Next, evaluating the derivative, x, we have d/dx r2 , and setting this to zero when x =  2 x∗ A∗ A − 2b∗ A = 0 or x = A∗ b (A∗ A)

(2.34)

Equation (2.34) is called the normal equation. The matrix A∗ A is also known as the x is unique if and only Grammian (or Gram matrix) of A.4 The solution of (2.34) for  if A∗ A is nonsingular. This condition is equivalent to having the columns of A be an independent set, as given in the following theorem: 4

Some textbooks refer to A∗ A as the normal matrix. However, the term normal matrix is used in other textbooks (as we do in this text) to refer more generally to matrices K that satisfy KK∗ = K∗ K.

72

Solution of Multiple Equations THEOREM 2.1.

The columns of A[=]N × M are linearly independent if and only if A∗ A

is nonsingular

PROOF.

The columns of A can be combined linearly as ⎞ ⎛ α1 M  ⎟ ⎜ α j A•,j = 0 → A ⎝ ... ⎠ = 0 j =1 αM

Multiplying by A∗ ,

⎞ α1 ⎟ ⎜ A∗ A ⎝ ... ⎠ = 0 αM ⎛

Thus, if A∗ A is nonsingular, the unique solution is given by α j = 0 for all j , that is, the columns of A are linearly independent if and only if A∗ A is nonsingular.

Returning to the optimization problem, we need to check if the minimum is achieved. The Hessian of r2 is given by   d2  2  d ∗ ∗ r = 2 (A A) x − A b = 2A∗ A dx2 dx Assuming that the columns of A are linearly independent, then A∗ A is positive definite.5 In summary, suppose the columns of A are linearly independent, then the least-squares solution of Ax = b is given by  x = (A∗ A)−1 A∗ b = A† b

(2.35)

where A† is the pseudo-inverse of A. Definition 2.2. Let A[=]N × M whose columns are linearly independent. Then the pseudo-inverse of A, denoted by A† , is defined as A† = (A∗ A)−1 A∗ If A is nonsingular, the pseudo-inverse is the same as the inverse of A, that is, A† = (A∗ A)−1 A∗ = A−1 (A∗ )−1 A∗ = A−1 Other properties of A† are: 1. AA† and A† A are symmetric 2. AA† A = A 3. A† AA† = A† 5

With y = Ax, x∗ A∗ Ax = y∗ y =

N  i=1

y¯ i yi ≥ 0

(2.36)

2.5 Least-Squares Solution

73

Table 2.2. Sample experimental data T (o F)

P(mm Hg)

C(mole fraction)

T (o F)

P(mm Hg)

C(mole fraction)

60.2 70.1 79.9 90.0 62.3 75.5 77.2 85.3

600 602 610 590 680 672 670 670

0.660 0.672 0.686 0.684 0.702 0.711 0.713 0.720

60.7 69.3 78.7 92.6 60.5 70.6 81.7 91.8

720 720 725 700 800 800 790 795

0.721 0.731 0.742 0.742 0.760 0.771 0.777 0.790

Remark: In MATLAB, the pseudoinverse is implemented more efficiently by the backslash ( \ ) operator, thus instead of x=inv(A’*A)*(A’*b), one can use the command: x=A\b. One application of the least-squares solution is to obtain a linear regression model for a multivariable function, that is, y = α1 v1 + · · · + αM vM

(2.37)

where y is a dependent variable and v j is the j th independent variable. Note that for the special case in which the elements of the matrices are all real, we can replace the conjugate transpose by simple transpose.

EXAMPLE 2.7. Suppose want to relate the effects of temperature and pressure on concentration by a linear model given by

C = αT + βP + γ

(2.38)

that would fit the set of experimental data given in Table 2.2. Based on the model given in (2.38), we can formulate the problem as Ax = b, with ⎛ ⎞ ⎛ ⎞ 0.660 60.2 600 1 ⎛ ⎞ α ⎜ 0.672 ⎟ ⎜ 70.1 602 1 ⎟ ⎜ ⎟ ⎜ ⎟ x=⎝ β ⎠ b=⎜ A=⎜ . ⎟ ⎟ .. . . .. .. ⎠ ⎝ ⎠ ⎝ .. . γ 0.790 91.8 795 1 Because exact equality is not possible, we try for a least-squares solution instead, that is, ⎛ ⎞ 0.0010 x = A† b = ⎝ 0.0005 ⎠ 0.2998 The linear model is then given by C = 0.0010T + 0.0005P + 0.2998

(2.39)

A plot of the plane described by (2.39) together with the data points given in Table 2.2 is shown in Figure 2.4.

74

Solution of Multiple Equations

C ( mole fraction ) 0.9

Figure 2.4. A plot of the least-squares model together with data points for Example 2.6.

0.8

0.7

100

600 o T( F) 80

700

60 800

P (mm Hg)

The least-squares method can also be applied to a more general class of models called the linear-in-parameter models. This is given by y (v) =

M 

αi f j (v)

(2.40)

j =1

where v1 , . . . , vK are the independent variables and y(v), f 1 (v) , . . . , f M (v) are linearly independent functions. Methods for determining whether the functions are linearly independent are given in Section B.8.

EXAMPLE 2.8. Consider the laboratory data given Table 2.3. The data are to be used to relate vapor pressure to temperature using the Antoine Equation given by:

log10 (Pvap ) = A −

B T +C

(2.41)

where Pvap is the vapor pressure in mm Hg and T is the temperature in ◦ C. First, we can rearrange (2.41) as T log10 (Pvap ) = −C log10 (Pvap ) + AT + (AC − B) We can then relate this equation with (2.40) by setting y = T log10 (Pvap ), f 1 = log10 (Pvap ), f 2 = T and f 3 (T, P) = 1, α1 = −C, α2 = A and α3 = AC − B. After applying the data from Table 2.3, ⎞ ⎞ ⎛ ⎛ 1.291 29.0 1 37.43 ⎛ ⎞ α1 ⎜ 1.327 30.5 1 ⎟ ⎜ 40.47 ⎟ ⎟ ⎟⎝ ⎜ ⎜ ⎠ = α ⎟ ⎟ ⎜ ⎜ .. .. .. .. 2 ⎠ ⎠ ⎝ ⎝ . . . . α3 404.29 3.063 132.0 1 The normal equation yields α1 = −222.5, α2 = 7.39, and α3 = 110.28, or in terms of the original parameters, A = 7.39, C = 222.5, and B = 1534. Figure 2.5 shows the data points together with the model (2.41).

2.5 Least-Squares Solution

75

Table 2.3. Raw data from vapor pressure experiment T (◦ C)

Pvap (mm Hg)

T (◦ C)

Pvap (mm Hg)

29.0 30.5 40.0 45.3 53.6 60.1 72.0 79.7

20 21 35 46 68 92 152 206

83.5 90.2 105.2 110.5 123.2 130.0 132.0

238 305 512 607 897 1092 1156

Another common situation for some least-squares problem Ax =lsq b is that they might be accompanied by linear equality constraints. Let the constraints be given by Cx = z

(2.42)

where A[=]N × M, C[=]K × M with K < M < N, and both A and C are full rank. Using Gauss-Jordan elimination on C, we can find Q and W such that   0[K×(M−K)] QCW = I[K] then based on (2.8), the solution to (2.42) is given by ⎛ ⎞ Qz ⎜ ⎟ x=W ⎝ ⎠ v

(2.43)

  where v is a vector containing (M − K) unknown constants. Let W = WL WR with WL[=]M × K and WR [=]M × (M − K). Then applying (2.43) to the leastsquares problem, ⎛ ⎞ Qz   ⎜ ⎟ Ax = A WL WR ⎝ ⎠ = b → v = (AWR )† (b − AWLQz) v AWLQz + AWR v

=

b

Pvap ( mm Hg )

1500

1000

Figure 2.5. Comparison of the Antoine model and raw data.

500

0 20

40

60

80 o

T ( C)

100

120

140

76

Solution of Multiple Equations Table 2.4. Vapor-liquid equilibrium data fL

fV

0 0.0718 0.1121 0.1322 0.1753 0.1983 0.2500

0 0.1700 0.2332 0.1937 0.2530 0.3636 0.3478

fL 0.2931 0.3190 0.3362 0.3937 0.4052 0.4483 0.5172

fV

fL

0.4506 0.5257 0.5217 0.4032 0.5968 0.6522 0.6759

fV

0.5690 0.6236 0.6753 0.7443 0.7902 0.9080 0.9167 1.0000

0.7549 0.8103 0.8142 0.8300 0.8972 0.9289 0.9802 1.0000

where (AWR )† is the pseudo-inverse of (AWR ). This result is then used to form the desired least-squares solution, ⎛ ⎞ Qz ⎜ ⎟ † x=W ⎝ (2.44) ⎠ where v = (AWR ) (b − AWLQz) v EXAMPLE 2.9. Suppose we want to fit a second-order polynomial model to relate liquid mole fraction f L to the vapor mole fraction f V ,

f V = α + βf L + γf L2

(2.45)

using data given in Table 2.4. The least-squares problem is then given by Ax = b, where ⎛ ⎛ ⎞ ⎞ 1 0 0 0 .. .. .. ⎜ ⎜ .. ⎟ ⎟ ⎛ ⎞ ⎜ ⎜ . ⎟ ⎟ . . . α ⎜ ⎜ ⎟ ⎟ 2 ⎟ ⎜ ⎟ ⎝ β ⎠ A=⎜ ⎜ 1 ( f L)n ( f L)n ⎟ b = ⎜ ( f V )n ⎟ x = ⎜ ⎜ . ⎟ ⎟ . .. .. γ .. ⎝ ⎝ .. ⎠ ⎠ . . 1 1 1 1 The physical constraints due to pure substances require that f L = 0 at f V = 0, and f L = 1 at f V = 1.6 The constraints are Cx = z, where     1 0 0 0 C= and z = 1 1 1 1 Using (2.44), we obtain

⎞ 0 x = ⎝ 1.6868 ⎠ −0.6868 ⎛

Thus the model that satisfies the constraints is given by f V = −0.6868 f L2 + 1.6868 f L However, if the equality constraints were neglected, a different model is obtained and given by f V = −0.6348 f L2 + 1.5997 f L + 0.0263 6

For azeotropic systems, more constraints may have to be included.

2.6 QR Decomposition

77

1

0.8 fV

Figure 2.6. Comparison of models using constraints (solid line), models without using constraints (dashed line), and the data points (open circles).

0.6

0.4

0.2

0 0

0.2

0.4

0.6

0.8

f

L

The plots in Figure 2.6 compare both models. Although they appear close to each other, the violation of the constraints by the second model may present some complications, especially when they are applied to a process simulator.

2.6 QR Decomposition In this section, we introduce another factorization of A known as the QR decomposition. This allows for an efficient solution of the least-squares problem.7 The QR decomposition of A is given by A = QR, such that the columns of Q are orthogonal to each other, that is, Q∗ Q = I, and R is an upper triangular matrix.8 Details of the QR algorithm are included in Section C.2.1 as an appendix. We could then apply this factorization to solve the normal equation as follows, A∗ Ax

=

(R∗ Q∗ ) (QR) x

=

R∗ Rx

=

A∗ b A∗ b

(2.46)

Because R is upper triangular, R∗ R is already a Choleski LU factorization of A∗ A, and a forward and backward substitution can be used to find the least-squares solution x. Remarks: 1. In MATLAB, QR decomposition can be obtained using the command [Q,R]=qr(A,0), where the option 0 will yield an “economy version” such that Q[=]N × M and R[=]M × M if A[=]N × M and N > M. However, if Q is not needed, as in (2.46), the command R=qr(A) will yield R if A is stored as a sparse matrix; otherwise, one needs to extract the upper triangular portion, that is, R=triu(qr(A)). 2. The QR factorization exists for all A regardless of size and rank. Thus it presents one method for the least-squares solution of Ax =lsq b in case A∗ A is singular.9 A MATLAB code that implements the QR algorithm is available on the book’s webpage as QR_house.m, where [R,P,Q]=QR_house(A) yields an 7 8 9

The QR algorithm is also an efficient method for calculating eigenvalues, as discussed in Section C.2. Another classic method for finding a set of orthogonal vectors that has the same span of a given set of vectors is the Gram-Schmidt method. Details of this method are given in Section B.9. Another method is to use SVD to find the Moore-Penrose inverse, (cf. Section 3.9.1).

1

78

Solution of Multiple Equations

additional permutation matrix P such that QR = AP, Q∗ Q = I and R is an upper triangular matrix whose diagonal is arranged in decreasing magnitude. This means that if A is not full rank, R can be partitioned to be  R=

 R 0

0 0



(see exercise E2.13, part b). In this case, due to the permutation P, the normal equation of (2.46) will have to be modified to become R∗ Rx = PT A∗ b If Q is not needed, then use instead [R,P]=QR_house(A).

2.7 Conjugate Gradient Method The conjugate gradient (CG) method is an iterative method for the solution of Ax = b, where A is a real symmetric positive definite matrix. Let x(i) be the ith update of x.10 The method is based on updating the value of x(i) such that the scalar function f (x) given by f (x) = xT Ax − xT b

(2.47)

is minimized. Let  x be the exact solution, then f ( x) = 0. Because the conjugate gradient method is iterative, it can take advantage of the sparsity in matrix A when evaluating matrix products. However, unlike the Jacobi, Gauss-Seidel, or SOR methods, the conjugate gradient method is guaranteed to reach the solution within a maximum of N moves assuming there are no round-off errors, where N is the size of x. If small roundoff errors are present, the Nth iteration should still be very close to  x. The conjugate gradient method is currently one of the more practical methods to solve linear equations resulting from finite-element methods. The resulting matrices from finite-element methods are usually very large, sparse, and, in some cases, symmetric and positive definite. In its simplest formulation, the method involves only a few calculations in each step. The update equations are given by x(i+1) = x(i) + α(i) d(i)

(2.48)

where α(i) and d(i) are the ith weight and ith update vector, respectively. The new value will have a residual error given by r(i+1) = b − Ax(i+1) (i)

d

10

(2.49)

The conjugate gradient method simply chooses the weight α(i) and update vectors such that  T d(i) Ar(j ) = 0 for j < i (2.50) We will use the un-bold letter xi for the ith element of x.

2.8 GMRES

that is, d(i) will be A-orthogonal, or conjugate, to the past residual vectors r(j ) , j < 1 (which also happens to be the negative gradient of f (x)). This criteria will be satisfied by choosing the following: ⎧ ⎪ (0) ⎪ for i = 0 ⎨ r (r(i) )T r(i) (i) d = and α(i) = (i) T (i) (2.51) (i) T (i−1) (r ) Ad ⎪ (d ) Ad ⎪ d(i−1) for i > 0 ⎩ r(i) − (d(i−1) )T Ad(i−1) The function f (x) will in general have ellipsoidal contours surrounding the origin. The criteria given in (2.50) is based on A-orthogonality. This just means that with A = ST S, the function f will have spherical contours under the new space y = Sx. In this new space, each iteration is orthogonal to past moves, yet are optimally directed to the exact solution from its current subspace. This makes the convergence quite fast. More importantly, all these moves happen without having to evaluate the factor S. The details of the conjugate gradient method are given in Section B.11 as an appendix. In that section, we include the theorems that prove the claim that criteria (2.50) is attained by using (2.51) and that the method converges in N moves (assuming no roundoff errors). Remarks: In MATLAB, the command for conjugate gradient method is given by x=pcg(A,b) to solve Ax = b. Also, a MATLAB function is available on the book’s webpage for the conjugate gradient method x=conj_grad(A,b,x0), where x0 is the initial guess.

2.8 GMRES The conjugate gradient method was developed to solve the linear equation Ax = b, where A is Hermitian and positive definite. For the general case, where A is nonsingular, one could transform the problem to achieve the requirements of the conjugate gradient method in several ways, including A∗ Ax = A∗ b or ⎛ ⎞⎛ ⎞ ⎛ ⎞ A ⎟⎜ r ⎟ ⎜ b ⎟ ⎜ I ⎜ ⎟⎜ ⎟=⎜ ⎟ ⎝ ∗ ⎠⎝ ⎠ ⎝ ⎠ A 0 x 0 Another approach, known as the Generalized Minimal Residual (GMRES) Method, introduces an iterative approach to solve Ax = b that updates the solution x(k) by reducing the norms of residual errors at each iteration. Unlike the conjugate gradient method, GMRES is well suited for cases with non-Hermitian matrix A. Briefly, in GMRES, a set of orthogonal vectors u(k) is built sequentially using another method known as Arnoldi’s algorithm, starting with u(0) = r(0) . At each introduction of u(k+1) , a matrix U k+1 is formed, which is then used to solve an associated least-squares problem ⎞ ⎛ r(0) ⎟ ⎜ 0   ∗ ⎟ ⎜ U k+1 AU k yk =lsq ⎜ ⎟ .. ⎠ ⎝ . 0

79

80

Solution of Multiple Equations

to obtain vector yk . This result is then used to solve for the kth estimate, x(k) = x(0) + U k yk

(2.52)

The process stops when the residual r(k) = b − Ax(k) has a norm less than a specified tolerance. Although it looks complicated, the underlying process can be shown to minimize the residual at every iteration. Thus the rate of convergence is quite accelerated. In some cases, the solution may even be reached in much fewer iterations than N. However, if round-off errors are present or if A is not well conditioned, the algorithm may be slower, especially when U k starts to grow too large. A practical solution to control the growth of U k is to invoke some restarts (with the current estimate as the initial guess at the restart). This would degrade the efficiency of the algorithm because the old information would be lost at each restart. Fortunately, the new initial guesses will always be better than the initial guesses of previous restarts. The details of GMRES, including some enhancements to accelerate the convergence further, can be found in Section B.12 as an appendix. Remarks: In MATLAB, the command for the GMRES method is given by x=GMRES(A,b,m) to solve Ax = b for x with restarts after every m iterations. Also, a MATLAB function gmres_method.m is available on the book’s webpage to allow readers to explore the algorithm directly without implementing restarts.

2.9 Newton’s Method Let us apply some of the methods of the previous sections to the solution of nonlinear equations. Newton’s method takes the linearization of nonlinear equations and converts the problem back to a linear one around a local iteration. Consider a set of n nonlinear functions, f i , of n variables: x1 , . . . , xn : ⎛ ⎞ f 1 (x1 , . . . , xn ) ⎜ ⎟ .. F(x) = ⎝ ⎠=0 . f n (x1 , . . . , xn ) Newton’s method is an iterative search for the values of x such that F(x) is as close to zero as the method allows. Using an initial guess, x(0) , the values of x(k+1) is updated from x(k) by adding a correction term k x, x(k+1) = x(k) + k x To determine the correction term, we can use the Taylor series expansion of F(x(k+1) ) around x(k) ,    dF     (k+1) (k)  x + ... F x(k+1) = F x(k) + − x dx x=x(k)   By forcing the condition that F x(k+1) = 0 while truncating the Taylor series expansion after the second term, we obtain the update equation for Newton’s method,     (2.53)

k x = x(k+1) − x(k) = −J k−1 F x(k)

2.9 Newton’s Method

where J k is the Jacobian matrix of F evaluated at x = x(k) , ⎛ ⎞ ∂ f1 ∂ f1 · · · ⎜ ∂x1 ∂xn ⎟ ⎜ ⎟  ⎜ .. ⎟ .. dF  ⎜ .. ⎟ . . ⎟ Jk = =⎜ . ⎟ dx x=x(k) ⎜ ⎜ ⎟ ⎝ ∂ fn ∂ fn ⎠ ··· ∂x1 ∂xn x=x(k)

  The updates are then obtained in an iterative manner until the norm of F x(k) is below a set tolerance, . In summary, the Newton’s method is given by the following procedure: Algorithm of Newton’s Method. 1. Initialize. Choose an initial guess: x(0)   2. Update. Repeat the following steps until either F x(k) ≤  or the number of iterations have been exceeded (a) Calculate J k . (If J k is singular, then stop the method and declare   “Singular Jacobian.”) (b) Calculate the correction term: k x = −J k−1 F x(k) (c) Update x: x(k+1) = x(k) + k x

Remarks: 1. In general, the formulation of the exact Jacobian may be difficult to evaluate. Instead, approximations are often substituted, including the simplest approach, called the secant method, which uses finite difference to approximate the partial derivatives, that is, ⎛ ⎞ ⎛ ⎞   (k)   (k)  ∂ f1 ∂ f1  · · · s1N x ⎜ ∂x1 · · · ∂xN ⎟ ⎜ s11 x ⎟ ⎜ ⎟ ⎜ ⎟  ⎜ .. ⎟ ⎜ ⎟ . . . . . .. .. ⎟ .. .. .. Jk = ⎜ . ≈⎜ ⎟ (2.54) ⎜ ⎟ ⎜ ⎟    ⎝ ∂ fN ⎝  (k)  ⎠ ∂ f N ⎠ (k) · · · sNN x sN1 x ···  (k) ∂x1 ∂xN x=x where sij (x) =

f i (x1 , . . . , x j + x j , . . . , xN ) − f i (x1 , . . . , xN )

x j

Because the approximation in (2.54) may often be computationally expensive, other approaches include a group of methods known as Quasi-Newton methods. One of the most popular is the Broyden method, in which the Jacobian J k is approximated by a matrix B(k) that is updated by the following rule ' ( 1 (k+1) (k) (k)

k F − B k x ( k x)T =B + B (2.55) T ( k x) k x where

   

k F = F x(k+1) − F x(k)

81

82

Solution of Multiple Equations

and B(0) can be initiated using (2.54). Moreover, because the Newtonupdate  in fact needs the inverse of the Jacobian, that is, k x = −J k−1 F x(k) ≈ −1  (k)   F x , an update of the inverse of B(k) is more desirable. Thus, − B(k) using (1.27), (2.55) can be inverted to yield ( −1  −1 1 '  −1  −1 = B(k) + B(k+1) B(k)

k F − k x k xT B(k) (2.56) α where  −1 α = k xT B(k)

k F 2. One can also extend Newton’s method (and its variants) to the solution of unconstrained minimization min f (x)

(2.57)

x

by setting  F(x) =

d f dx

T

 and

J (x) =

d2 f dx2

 (2.58)

where J (x) is the Hessian matrix of f (x). The point x∗ becomes a minimimum of f (x) if (df/dx)T (x∗ ) = 0 and d2 f/dx2 (x∗ ) > 0. One practical concern with Newton methods is that the convergence to the solutions are strongly dependent on the initial guesses. There are several ways to improve convergence. One approach is known as the line search method, also known as the back-tracking method, which we discuss next. Another approach is the doubledogleg method, and the details of this approach are given in Section B.13 as an appendix. Remarks: In MATLAB, the command to find the solution of a set of nonlinear equation is fsolve. Also, another function is available for nonlinear least squares given by lsqnonlin.

2.10 Enhanced Newton Methods via Line Search Consider the scalar function f (x) = tanh(x) whose root is given by x = 0. Starting with an initial guess x0 = 0.9, the left plot in Figure 2.7 shows that Newton’s method converges very close to the root after three iterations. However, if we had chosen x0 = 1.1 as the initial guess, the right plot in Figure 2.7 shows that the method diverges away from the root. This shows that the success of Newton’s method is highly dependent on the initial guess. However, it also shows that if only a fraction of the correction step k x had been taken, the next few iterations may move the value of x into a region in which the Newton method is convergent. Instead of (2.53), let the update be

k x = λδk

(2.59)

2.10 Enhanced Newton Methods via Line Search 1

1 ( x , f( x ) )

df (x ) dx k

0.5

f(x)

x

0

k

0.5 x

k+1

k

f(x)

k

−0.5

−1 −2

83

0

xk+1

x

k

xk+2

−0.5

−1

0

1

2

x

−1 −2

−1

0

x

1

Figure 2.7. The performances of the Newton’s method to finding the solution of tanh(x) = 0. The left plot used x0 = 0.9 as the initial guess, whereas the right plot used x0 = 1.1 as the initial guess.

where

  δk = −J k−1 F x(k)

(2.60)

If λ is chosen too small, the number of iterations may become unnecessarily large. On the other hand, with λ close to 1, we end up with the same Newton update. To determine a good value of λ, we first define the scalar criterion function   2 1 φk (λ) = F x(k) + λδk (2.61) 2 and search for the value of λ that would minimize φk (λ). However, we will use polynomial curve fits instead of using Newton’s method. We will denote the line search procedure by searching for a sequence {λ0 , λ1 , . . . , λm , . . . , λ }, which terminates when F (xk + λ δk ) is acceptable. The conditions for acceptance of λ is given later in (2.67). We first discuss the calculations for λm , m = 0, 1, . . . , . The initial value of the line search, λ0 , can be found by using three terms:   2 1 F x(k) φk (0) = 2    d   φk (0) = φk  = FT x(k) J k δk dλ λ=0   2 1 F x(k) + δk φk (1) = 2 These terms determine a polynomial approximation of φk (λ) given by       P0 (λ) = φk (1) − φk (0) − φk  (0) λ2 + φk  (0) λ + φk (0)

(2.62)

Setting the derivative of (2.62) equal to zero, we obtain the value λ0 that would minimize P0 , φk  (0)  λ0 = −  2 φk (1) − φk (0) − φk  (0)

(2.63)

2

84

Solution of Multiple Equations

If the λ0 is not acceptable, we continue with an iteration scheme for λm , m = 1, 2, . . ., and use the values of φk at two previous values of λ, that is, φk (λm−1 ) and φk (λm−2 ), together with φk (0) and φk  (0). These four terms now determine a unique cubic polynomial given by     (2.64) Pm (λ) = aλ3 + bλ2 + φk  (0) λ + φk (0) where a and b can be obtained by solving ⎞ ⎛ ⎛ ⎞⎛  3 2 ⎜ λm−1 λm−1 ⎟ ⎜ a ⎟ ⎜ φk (λm−1 ) − φk (0) λm−1 − φk (0) ⎟=⎜ ⎜ ⎟⎜ ⎠ ⎝ ⎝ 3 ⎠⎝ b λm−2 λ2m−2 φk (λm−2 ) − φk  (0) λm−2 − φk (0) The minimum of Pm (λ) can be found to be ! −b + b2 − 3aφk  (0) λm = 3a

⎞ ⎟ ⎟ ⎠

(2.65)

(For m = 1, we set λm−2 = λ−1 = 1 ). Although (2.63) and (2.65) generate the minima based on their corresponding polynomials, the fractional reduction needs to be controlled within a range, which is usually set as   1 1 λm ≤ ≤ (2.66) 10 λm−1 2 This is to avoid producing a λ that is too small, which could mean very slow convergence of the line search updates or even a premature termination of the solution. On the other hand, if the decrease of λ is too small, the line search updates are more likely to miss the regions that accept regular Newton updates. The remaining issue is the acceptability condition for λ. A simple criteria is that an acceptable λ = λ occurs when the average rate of change of φk is at least a fraction of the initial rate of change φk  (0), for example, for α ∈ (0, 1], φk (λ ) − φk (0) ≤ αφk  (0) λ or equivalently, φk (λ ) ≤ φk (0) + α λ φk  (0)

(2.67)

It can be shown that with α ≤ 0.25, (2.65) can be guaranteed to have real roots. However, a usual choice is to set α as low as 10−4 . Because λ ≤ 1, this parameter is often referred to as the damping coefficient of the line method. To summarize, we have the following enhanced Newton line search procedure: Algorithm of Enhanced Newton’s Method with Line Search. 1. Initialize. Choose an initial guess: x(0)   2. Update. Repeat the following steps until either F x(k) ≤  or the number of iterations have been exceeded. (a) Calculate J k . (If J k is singular, then stop the method anddeclare  “Singular Jacobian.”) (b) Calculate the correction term: δk = −J k−1 F x(k)

2.10 Enhanced Newton Methods via Line Search

85

1 1

0.5

f(x)

0.5

f(x)

0 x

xk

k+1

0

xk+1

x

xk+2

k

−0.5

−0.5

−1

−1 −2

−1

0

x

1

2

−4

−2

0

2

x

Figure 2.8. The performances of the line search method to finding the solution of tanh(x) = 0. The left plot used x0 = 1.1 as the initial guess, whereas the right plot used x0 = 4.0 as the initial guess.

(c) Solve for damping coefficient: Evaluate λm , (m = 0, 1, 2, . . .) using (2.63) for m = 0 and (2.65) for m > 0, but clipped according to the range of ratios given in (2.66), until m = , where λ satisfies (2.67) (d) Update x(k) : x(k+1) = x(k) + λ δk Remarks: A MATLAB code for the enhanced Newton method is available on the book’s webpage as nsolve.m, and the line search method is implemented when the parameter “type” is set to 1. Also, a MATLAB file NewtonMin.m is available on the book’s webpage that uses the ehanced Newton method for minimization of a scalar function, where the line search method is implemented when the parameter type is set to 1. In Figure 2.8, we see how the line search approach improves Newton’s method. On the left plot, the initial guess of x0 = 1.1 using Newton’s method had yielded a divergent sequence (cf. Figure 2.7), but with the inclusion of line search, it took about one iteration to yield a value that is very close to the final solution. The right plot shows that even using an initial guess of x0 = 4.0 where the slope is close to flat, the line search method reached a value close to the solution in about two iterations. There are other alternative search directions to (2.60). These directions can be used when the Jacobian J k = dF/dxk is singular, or is close to being singular. These search directions are: 1. Gradient search: δk = −J kT F  −1 T 2. Marquardt search: δk = − J kT J k + μI J k F, where μ is chosen such that |δk | ≤ β, β > 0. † † 3. Pseudoinverse Search: δk = −J k F where J k is the pseudo-inverse of the Jacobian. The last two alternatives are often used when the dimension of F is much higher than the dimension of x, yielding a nonlinear least-squares problem. The LevenbergMarquardt is a method for nonlinear least squares that combines the gradient search and the Newton search directions. Details of this method are included in Section B.14 as an appendix.

4

86

Solution of Multiple Equations 2.11 EXERCISES

E2.1. The reaction for a multiple reversible first-order batch reaction shown in Figure 2.9,

Figure 2.9. A three-component reversible reaction.

is given by dx = Ax dt where

⎛

−(kab + kac ) kab A=⎝ kac

kba −(kba + kbc ) kbc

⎞ kca ⎠ kcb −(kca + kcb)

⎛

;

⎞ xa x = ⎝ xb ⎠ xc

xi is the mass fraction of component i and kij is the specific rate constant for the formation of component j from component i. The equilibrium is obtained by setting dx/dt = 0, i.e. A xeq = 0. 1. Show that A is singular. 2. Because A is singular, the linear equation should yield multiple equilibrium values, which is not realistic. Is there a missing equation? If so, determine the missing equation. 3. Is it possible, even with the additional equation, that a case of non-unique solution or a case with no solution may occur? Explain. E2.2. Let A[=]N × N, B[=]M × M, G[=]N × N, H[=]M × M and C[=]N × M by given matrices. Let X[=]N × M be the unknown matrix that must satisfy AX + XB + GXH = C Obtain the conditions for which the solution X will be unique, non-unique, or nonexisting. E2.3. Find X (possibly infinite solution) that satisfies the following equation: ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ 3 2 1 1 2 3 9 16 23 ⎝ 4 5 6 ⎠ X + X ⎝ 0 4 5 ⎠ = ⎝ 23 18 32 ⎠ 1 1 1 1 6 8 8 22 31 using the matrices Q and W found based on: 1. Gauss-Jordan method 2. Singular Value Decomposition method E2.4. Consider the network given in Figure 2.10, given S1 = S3 = 15 v, A4,5 = 10 mA, and all the resistors have 1 K except for R6 = 100 . Based on the notations and equations given in Example 1.6,

2.11 Exercises R2

R3

1

2

R1 S1 -

+ S 3

R6

R7 R9

R8

R 11

R 10

0

3

R5

R4 +

87

4

5

A4,5 Figure 2.10. Resistive network containing two voltage sources and one current source.

we have    −1 T  R  p = b − R−1 s where p is the vector of node potentials,  is the node-link incidence matrix, p is the vector of current sources, s is the vector of voltage sources, and R is a diagonal matrix consisting of the resistance of each link. Use the LU factorization approach to solve for p, and then solve for the voltages v across each resistor, where v = −T p − s E2.5. Redo the boundary value problem given in Example 2.4 but using the following functions: h2 (x) = 2 + x , h1 (x) = 5x , h0 (x) = 2 + x     f (x) = −5x2 + 11x − 8 e−2x+6 + 8x2 + 57x + 2 and determine u(x) for 0 ≤ x ≤ 10 with the boundary conditions u(0) = 1 and u(10) = 81. Use the finite interval x = 0.01 and compare with the exact solution, u(x) = (1 + 8x) + xe−2x+6 (Note: Finding the exact solution for a high-order linear differential equation containing variable coefficients is often very difficult. We have an exact solution in this case because we actually obtained the differential equation and the boundary conditions using u(x).) E2.6. Let matrix M be a tri-diagonal matrix plus two additional nonzero elements m1,3 = s and mN,N−2 = t, ⎛ ⎞ a1 b1 s 0 ⎜ ⎟ .. .. ⎜ c1 ⎟ . . ⎜ ⎟ ⎜ ⎟ . . . .. .. .. (2.68) M=⎜ ⎟ ⎜ ⎟ ⎜ ⎟ .. .. ⎝ . . bN−1 ⎠ t cN−1 aN

88

Solution of Multiple Equations

1. Using Crout’s method (without pivoting), M can be decomposed as M = LU, where ⎛ ⎞ z1 0 ⎜ c1 z2 ⎟ ⎜ ⎟ ⎜ ⎟ . . .. .. L=⎜ ⎟ ⎜ ⎟ ⎝ 0 ⎠ cN−2 zN−1 0 t N,N−1 N,N ⎛ ⎞ s 1 f1 0 ⎜ ⎟ a1 ⎜ ⎟ ⎜ 1 f2 0 ⎟ ⎜ ⎟ U =⎜ (2.69) ⎟ .. .. ⎜ ⎟ . . ⎜ ⎟ ⎝ 1 f N−1 ⎠ 0 1 with z1 = a1 fk =

zk = ak − ck−1 f k−1 , k = 2, . . . , N

;

bk , for i = 1 . . . , N − 1, i = 2 zk

f2 =

;

b2 s c1 − z2 z2 a1

and N,N−1 = cN−1 − tf N−2

N,N = zN + tf N−1 f N−2

;

Verify that LU = M. (Note that L and U are almost bi-diagonal, i.e., except for the additional nonzero terms: N,N−2 = t and u1,3 = s/a1 .) 2. Develop a modification to the Thomas algorithm that implements the forward and backward substitution using the L and U defined in (2.69) to solve for Mx = v. (Check to see that your equations reduces to the original Thomas algorithm if s = t = 0.) E2.7. Consider the equation ⎛ 4 −1 ⎜ −1 4 ⎜ ⎜ −1 −1 ⎜ ⎜ ⎜ 0 Ax = ⎜ −1 ⎜ . .. .. ⎜ . ⎜ ⎜ . .. .. ⎝ . −1 0

−1 −1 4

−1 0 −1

−1 .. .

4 .. . ..

···

. ···

−1 0 0 .. .

··· ··· ··· .. .

..

..

..

.

. 0

..

.

. −1

−1 0 0 .. .

⎞

⎟ ⎛ ⎟ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎟x = ⎜ ⎜ ⎟ ⎝ 0 ⎟ ⎟ ⎟ −1 ⎠ 4

1 2 .. .

⎞

⎟ ⎟ ⎟ ⎟ ⎟ 2 ⎠ 1

where A[=]50 × 50. 1. Use LU factorization of A to solve for x. ( Note that the L matrix has several fill-ins.) 2. If we permute the rows and columns of A using row permutation P with the sequence [2, . . . , 50, 1] such that  A = PAPT , and then obtain the LU  factorization of A, what happens to the amount of fill-ins? Describe how one could use this fact to solve for x.

2.11 Exercises

89

E2.8. Use the direct matrix splitting method given in Sections 2.3, B.6 to solve the following problem: ⎛ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎝

1 −2 1 3 4 3

1 2 0 0 0 0

−1 2 2 0 0 0

0 5 −2 −3 0 0

1 1 1 −1 1 0

3 3 −2 1 2 1

⎛

⎞

⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎟x = ⎜ ⎜ ⎟ ⎜ ⎟ ⎝ ⎠

6 8 2 3 7 4

⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠

by using a split A = M + S, with M containing the upper triangular portion of A. (Note: Based on the equations given in Section 2.3, note that one can use backward substitution when evaluating M−1 S or M−1 b, i.e., there is no need to explicitly find the inverse of M. This is automatically done in MATLAB when using the backslash (’ \ ’) or mldivide operator.) * ) E2.9. For a given data table (x1 , y1 ) , . . . , (xN , yN ) , with xk+1 > xk , cubic spline interpolation generates a set of piecewise-continuous functions that passes through each data point using cubic polynomials. In addition, the interpolating function can be made to have continuous first- and second-order derivatives at each data point. For x ∈ [xk , xk+1 ], a polynomial defined as ⎛ 

p k (x) = yk+1

yk

γk+1

x3k+1

⎜  ⎜ x2k+1 γk ⎜ ⎜x ⎝ k+1 1

x3k

6xk+1

x2k

2

xk

0

1

0

6xk

⎞−1 ⎛

⎟ 2 ⎟ ⎟ 0 ⎟ ⎠ 0

x3

⎞

⎜ 2⎟ ⎜x ⎟ ⎜ ⎟ (2.70) ⎜x⎟ ⎝ ⎠ 1

will satisfy the following conditions: p k (xk ) = yk ; p k (xk+1 ) = yk+1

  d2 p k  d2 p k  ; = γk ; = γk+1 dx2 x=xk dx2 x=xk+1

By specifying continuity plus continuous first- and second-order derivatives between pairs of connecting polynomials, it can be shown that the following conditions result: 

xk γk−1 + 2 ( xk + xk+1 ) γk + xk+1 γk+1 = 6

yk+1

yk −

xk+1

xk

 (2.71)

where xk = xk − xk−1 and yk = yk − yk−1 . Equation (2.71) applies only to k = 2, . . . , N − 1. Two additional specifications are needed and can be obtained by either setting (i) γ1 = 0 and γN = 0 (also known as the natural condition), or (ii) γ1 = γ2 and γN = γN−1 , or (iii) setting γ1 , γ2 , and γ3 to be collinear and γN−2 , γN−1 , and γN to be collinear, that is, γ3 − γ1 γ 3 − γ2 = x3 − x1 x3 − x2

and

γN−2 − γN γN−2 − γN−1 = xN−2 − xN xN−2 − xN−1

(2.72)

90

Solution of Multiple Equations

Show that (2.71) can be combined with the various types of end conditions to give the following matrix equation: ⎛ ⎞ 0 ⎜ ⎟   ⎟ ⎛ ⎞ ⎜ ⎜ ⎟

y

y γ1 3 2 ⎜ ⎟ ⎜ ⎟ ⎜ 6 x3 − x2 ⎟ ⎜ . ⎟ ⎜ ⎟ ⎜ . ⎟ ⎜ ⎟ . .. (2.73) M⎜ . ⎟ = ⎜ ⎟ ⎜ ⎟ ⎜   ⎟ ⎟ ⎝ ⎠ ⎜ ⎜ 6 yN − yN−1 ⎟ ⎜ γN

xN

xN−1 ⎟ ⎜ ⎟ ⎝ ⎠ 0 where

⎛

a ⎜ x2 ⎜ ⎜ M=⎜ ⎜ ⎝

b 2 ( x2 + x3 ) .. .

0

c

x3 .. .

xN−1 d

0 ..

. 2 ( xN−1 + xN ) e

xN f

⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠

and End conditions

Parameters

γ1 = 0

(a, b, c) = (1, 0, 0)

γN = 0

(d, e, f ) = (0, 0, 1)

γ1 = γ2

(a, b, c) = (1, −1, 0)

γN−1 = γN

(d, e, f ) = (0, −1, 1)

γ1 , γ2 , γ3 Collinear

(a, b, c) = ( x3 , − ( x2 + x3 ) , x2 )

γN−2 , γN−1 , γN Collinear

(d, e, f ) = ( xN , − ( xN−1 + xN ) , xN−1 )

Note that matrix M is tri-diagonal for the first two types of end conditions, but the third type (i.e., with collinearity) has the form given in (2.68). Thus (2.73) can be used to generate the values of second derivatives γk , k = 1, . . . , N, which can then be substituted into (2.70) in the appropriate interval [xk , xk+1 ]. Using these results, obtain a cubic spline curve that satisfies (2.72) and passes through the data points given in Table 2.5. Table 2.5. Data for spline curve interpolation x −0.9931 −0.7028 −0.4908 −0.3433 −0.1406 −0.0161 0.2143 0.4355 0.6244 1.0023

y 0.9971 1.4942 1.3012 0.9094 1.0380 1.3070 1.4415 1.1023 0.8509 0.6930

2.11 Exercises

91

E2.10. The 2D simple linear regression model is given by z = mw + c where z and w are the independent and dependent variables, respectively, m is the slope of the line, and c is the z-intercept. Show that the well-known regression formulas N N  N N i=1 wi zi − i=1 wi i=1 zi m = N 2 N N i=1 wi − ( i=1 wi )2 N c

=

N

N i=1 wi i=1  2 ( N w ) i i=1

N

2 i=1 wi − N 2 N i=1 wi −

i=1 zi

wi zi

can be obtained from the solution of the normal equation: AT Ax = AT b where

⎛

w1 ⎜ .. A=⎝ . wN

⎞ 1 .. ⎟ . ⎠ 1

 x=

⎛

⎞ z1 ⎜ ⎟ b = ⎝ ... ⎠ zN



m c

E2.11. Using data given in Table 2.6, obtain the parameters α1 , α2 , and α3 that would yield the least-squares fit of the following model: z = (α1 w2 + α2 w + α3 ) sin(2πw)

(2.74)

Table 2.6. Data to be fit by (2.74) w

z

w

z

w

z

w

z

w

z

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

0.65 −0.01 0.87 −0.55 −1.02 −0.46 −0.08 −1.23 −1.99 0.89

1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2.0

1.82 3.15 4.01 2.51 −0.21 −3.39 −8.18 −7.52 −4.23 0.15

2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 3.0

7.12 11.25 14.27 9.11 −1.81 −9.44 −18.24 −16.55 −11.20 0.64

3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 4.0

13.99 24.27 23.53 15.45 0.62 −16.37 −30.67 −31.63 −20.13 −2.48

4.1 4.2 4.3 4.4 4.5 4.6 4.7 4.8 4.9 5.0

23.33 36.52 40.41 24.56 0.63 −26.76 −46.66 −47.83 −31.23 0.55

E2.12. Using the vapor liquid equilibrium data shown in Table 2.7, obtain a 5th order polynomial fit of f V as a function of f L that satisfies the following constraints: fV = 0 fV = 1 f V = 0.65

when f L = 0 when f L = 1 when f L = 0.65

Table 2.7. Vapor liquid equilibrium data fL

fV

fL

fV

fL

fV

fL

fV

0.02 0.06 0.11 0.18 0.22

0.07 0.16 0.24 0.33 0.35

0.29 0.37 0.50 0.71 0.78

0.43 0.50 0.56 0.68 0.74

0.85 0.89 0.95 0.99 0.25

0.79 0.86 0.92 0.97 0.37

0.38 0.54 0.81 0.94 0.97

0.45 0.58 0.73 0.88 0.93

92

Solution of Multiple Equations

E2.13. Let A = QR be the QR factorization of A[=]N × M described in Section 2.6. It was mentioned that R could immediately be used as the Choleski factor in solving RT Rx = AT b, that is, forward and backward substitution can be used to find x using R. Of course when N is small, for example, 2 or 3, it appears that this approach is unnecessarily complicated because the inverse of AT A may be determined just as fast. However, if AT A is large and singular, the QR factorization may offer a viable alternative. 1. Solve to following least-squares problem: ⎞ ⎛ ⎞ ⎛ 3 1 5 ⎜ 4 1 ⎟ ⎜ 6 ⎟ ⎟ ⎜ ⎟ ⎜ ⎜ 5 1 ⎟ ⎟ ⎜ ⎟ x =lsq ⎜ 7 ⎟ ⎜ ⎜ 6 1 ⎟ ⎜ 8 ⎟ ⎟ ⎜ ⎟ ⎜ ⎝ 7 1 ⎠ ⎝ 9 ⎠ 8 1 10 use R obtained from the QR factorization of A and solve for x. Compare this to using  R obtained from the Choleski factorization of AT A, that is, T T  R  R = A A. 2. Try the same approach on the following problem: ⎞ ⎛ ⎞ ⎛ 5 3 1 2 ⎜ 6 ⎟ ⎜ 4 1 3 ⎟ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎜ 5 1 4 ⎟ ⎟ x =lsq ⎜ 7 ⎟ ⎜ (2.75) ⎜ 8 ⎟ ⎜ 6 1 5 ⎟ ⎟ ⎜ ⎟ ⎜ ⎝ 9 ⎠ ⎝ 7 1 6 ⎠ 10 8 1 7 Is a situation possible in which there are no least-squares solution? If solutions exist, how would one find the infinite least-squares solution x? Test your approach on (2.75). E2.14. A generalization of the Euclidean norm for a vector v is the Q-weighted norm (also known as the Riemanian norm), denoted by vQ , where Q is a Hermitian positive-definite matrix, defined as ! vQ = v∗ Qv (2.76) The Euclidean norm results by setting Q = I. 1. Show that the function •Q satisfies the conditions of norms as given in Table B.4. (Hint: With Choleski factorization, there exists a nonsingular S such that Q = S∗ S. Then use vQ = Sv and the fact that Euclidean norms satisfy the conditions for norms.) 2. Using r2Q , where (r = b − Ax) is the residual, show that the weighted least-squares solution is the solution to the weighted normal equation AT QAx = AT Qb

(2.77)

3. One application for the inclusion of variable weights is to weaken the influence of older data. Suppose rk is kth element of the residual vector, and rk is a more recent data than rk−1 . Next, choose Q to be ⎛ N−1 ⎞ λ 0 ⎜ ⎟ .. ⎜ ⎟ . (2.78) Q=⎜ ⎟ ⎝ ⎠ λ 0 1

2.11 Exercises

93

where 0 < λ < 1 is known as the forgetting factor. Then Qr attaches heavier weights on more recent data. Using data given in Table 2.8 and (2.77), obtain a weighted linear regression model z = mw + c for λ = 1, 0.5, 0.1, and plot the three cases together with the data. Explain the effects of different λ choices. Table 2.8. Data for weighted linear regression z

w

z

w

z

w

0.0104 0.0173 0.0196 0.0403 0.0472 0.0518 0.0588 0.0726 0.0841 0.1094

0.0219 0.0921 0.1652 0.1827 0.3085 0.2588 0.1915 0.3933 0.3289 0.4430

0.1187 0.1671 0.1717 0.1970 0.2293 0.2477 0.3145 0.3353 0.3744 0.3952

0.5219 0.5892 0.5278 0.5892 0.6184 0.6652 0.7032 0.7529 0.7208 0.7880

0.4459 0.4896 0.5012 0.5472 0.6048 0.7039 0.7730 0.8306 0.9528 0.9781

0.7792 0.8465 0.8319 0.8728 0.8728 0.9167 0.9313 0.9459 0.9722 0.9956

E2.15. The least-squares solution for Ax = b can also be implemented recursively; that is, as more data come in, the normal equation is resolved. Instead of having to invert AT A at each arrival of a new set of data, we can implement a recursive least-squares approach. −1  After using N data, suppose we have stored the values of ZN = ATN AN and wN = ATN bN . When the new data come in, the data matrix and target vector become     AN bN bN+1 = AN+1 = vTN βN 1. Show that wN+1 = ATN+1 bN+1 = wN + βN vN 2. Using the Woodbury formula (cf. (1.27))11 , show that −1  1 = ZN − ZN+1 = ATN+1 AN+1 ZN vN vTN ZN T 1 + vN ZN vN

(2.79)

(2.80)

3. Use (2.79) and (2.80) to obtain the correction fN in the following recursive formula: xN+1 = xN + fN

(2.81)

th

where xk is the k least-squares estimate of x. 4. Using QN as given in (2.78) with a fixed forgetting factor λ, determine how the recursive formulas can be modified to obtain a recursive weighted least-squares method. 11

For the inverse of (A + BCD), the Woodbury formula applied to the special case where B[=]N × 1 and C[=]1 × N is more well known as the Sherman-Morrison-Woodbury formula.

94

Solution of Multiple Equations

Note: The recursive least-squares method, with or without forgetting factors, is used for adaptive estimation of processes with variable parameters, for example, fouling, catalyst poisoning, or wear-and-tear. E2.16. In some cases, the linear equations will result in block tri-diagonal structures, for example, the open-loop process units in Example 1.11. Let the complete equation be given by Gx = v where G is block tri-diagonal ⎛ A1 ⎜ C1 ⎜ ⎜ G=⎜ ⎜ ⎝

B1 A2 .. .

⎞

0 B2 .. . CN−2

..

.

AN−1 BN−1 0 CN−1 AN where An , Bn , and Cn are all K × K block matrices.

⎟ ⎟ ⎟ ⎟ ⎟ ⎠

1. Show that the block-LU decomposition of G is given by G = LU where ⎛

Q1 ⎜ C1 ⎜ L=⎜ ⎝

0 Q2 .. .

..

. 0 CN−1 and, with C0 = 0, W0 = 0,

⎞

⎛

⎟ ⎟ ⎟ ⎠

⎜ ⎜ U=⎜ ⎝

QN

I

W1 .. .

0 ..

.

I 0

Qn

=

An − Cn−1 Wn−1

[=] K × K

Wn

=

Q−1 n Bn

[=] K × K

WN−1 I

⎞ ⎟ ⎟ ⎟ ⎠

2. Instead of evaluating Qn , the inverse, Zn = Q−1 n , will be of importance. Show that Z1

=

A−1 1

Zn

=

(An − Cn−1 Zn−1 Bn−1 )−1

n = 2, 3, . . . , N

3. Returning to the main problem of solving Gx = v and assuming all Zn exist, both x and v will have to be partitioned accordingly, that is, ⎞ ⎛ ⎞ ⎛ v1 x1 ⎟ ⎜ ⎟ ⎜ v = ⎝ ... ⎠ x = ⎝ ... ⎠ xN vN where xn and vn are vectors of length K. Having both L and U be block-triangular matrices, show that the forward substitution becomes y1

=

Z1 v1

yn

=

Zn (vn − Cn−1 yn−1 )

for n = 2, . . . , N

whereas the backward substitution becomes xN

=

yN

xn

=

yn − Zn Bn xn+1

for n = (N − 1) , . . . , 1

2.11 Exercises

95

Note: This the block-matrix version of the Thomas algorithm (cf. Section 2.2.2). If storage is a significant issue, one could use the storage of An and replace it with Zn , except for the case of patterned An ’s, as in the exercise that follows. Furthermore, from the backward substitutions given previously, there is no need to store Wn either. 4. For n = 1, . . . , 10, let       −4 1 −1 0 1 0 Bn = Cn = An = 1 −4 0 1 0 −1 and

 v1 =

2 2



 v10 =

3 1



 vk =

0 0

 for k = 2, 3, . . . , 9

Use the block version of the Thomas algorithm to solve for x. E2.17. When solving for the steady-state solution of an elliptic partial differential equation, the resulting linear equations will involve large sparse matrices. As an example, one could have the following: ⎛ ⎞ R −I 0 ⎜ −I ⎟ R −I ⎜ ⎟ ⎜ ⎟ . . . .. .. .. A=⎜ ⎟ ⎜ ⎟ ⎝ −I R −I ⎠ 0 −I R with

⎛

4 ⎜ −1 ⎜ ⎜ R=⎜ ⎜ ⎝

−1 4 .. .

0

−1 .. . −1

0

⎞

⎟ ⎟ ⎟ ⎟ ⎟ −1 ⎠ 4

..

. 4 −1

Let R[=]25 × 25 and A[=]625 × 625. Solve the linear equation Ax = b for x, when bk = 10 when k = 1, . . . , 5 and bj = 0 for j = 6, . . . , 625 using: 1. LU decomposition followed by forward and backward substitution. 2. Conjugate gradient method. 3. Block-Thomas algorithm (cf. see Exercise E2.16) E2.18. A hot liquid stream entering the annulus of a double-pipe heat exchanger at a temperature T hot,in = 300◦ C is being cooled by cold water counter-flowing through the inner pipe and entering at T cold,in = 25◦ C from the other side of the pipe. The working equations relating the temperatures of all the entering and exiting flows are given by α

(T hot,in − T cold,out ) − (T hot,out − T cold,in )   T hot,in − T cold,out ln T hot,out − T cold,in β (T hot,in − T hot,out )

=

T cold,out − T cold,in

=

(T cold,out − T cold,in )

where the parameters are α=

U inner (πDinner L) = 0.3 m ˙ cold Cp,cold

and

β=

m ˙ hot Cp,hot = 0.25 m ˙ cold Cp,cold

96

Solution of Multiple Equations

with U as heat transfer coefficient, D is the diameter, L is the length of the ˙ are the mass flow rates. pipes, Cp are the heat capacities, and m Let x = T hot,out and y = T cold,out 1. Solve for x and y by finding the roots of the following functions: (T hot,in − y) − (x − T cold,in )   − (y − T cold,in ) T hot,in − y ln x − T cold,in

f 1 (x, y)

=

α

f 2 (x, y)

=

β (T hot,in − x) − (y − T cold,in )

Apply Newton’s formula using the secant approximation for the Jacobian by choosing δx = 0.1 and δy = 0.1, ⎛ ⎞ ⎛ 1 ⎞ 0 f (x + δx, y) − f (x, y) f (x, y + δy) − f (x, y) 1 1 1 ⎜ 1 ⎟ δx ⎟ 1 ⎠⎜ J ≈⎝ ⎝ ⎠ 0 (x + δx, y) − f (x, y) f (x, y + δy) − f (x, y) f 2 2 2 2 δy and initial guess x = y = 325/2. 2. If one or more equation can easily be solved explicitly, this can improve computational efficiency by reducing the dimension of the search space. For instance, one could first solve for y in terms of x in f 2 (x, y) = 0 to obtain y = T cold,in + β (T hot,in − x) which can then be substituted into f 1 (x, y) to yield   f 3 (x) = f 1 x, T cold,in + β (T hot,in − x) Use Newton’s method using the secant approximation for the Jacobian like before, with the same initial guess x = 325/2. (Note: In this case, f 3 (x) is a scalar function, and one can plot the function to easily determine whether there are multiple roots.) E2.19. Given R equilibrium reactions that involves M > R components. Let vij be the stoichiometric coefficients of component j in reaction i and stored in a stoichiometric matrix V [=]R × M where ⎧ ⎨> 0 if component j is a product in (forward) reaction i vij = < 0 if component j is a reactant in (forward) reaction i ⎩ = 0 if component j is not part of reaction i After a sufficient time and under constant operating temperature, all the reactions will attain equilibrium such that the mole fractions will satisfy Ki =

M 

vj

xj i

j =1

where Ki is the equilibrium constant for the ith reaction. A typical problem is to determine the mole fractions x j based on a given set of initial moles n 0j for each component and equilibrium constants Ki . One effective approach to solve this problem is to use the concept of the extent of reaction ξi , which is the number of units that the ith reaction had undergone in the forward direction. For instance, let ξ2 = 2.5 and stoichiometric coefficient v23 = 2;

2.11 Exercises

97

then component 3 will gain ξ2 v23 = 5 moles due to reaction 2. Thus if we let n j be the number of moles of component j at equilibrium, we have n j = n 0j +

R 

vkj ξk

k=1

which allows the mole fractions at equilibrium to be defined in terms of ξk , that is,  nj n 0j + R k=1 vkj ξk x j = M = M R j =1 n j j =1 n 0j + k=1 βk ξk where βk =

M 

vkj

j =1

The problem then becomes finding the roots ξk , k = 1, . . . , R, of a vector of functions, f = ( f 1 , . . . , f R )T = 0 given by ⎞ ⎛  ⎛ vi j ⎞ R M M   n + v ξ 0j vj k=1 kj k ⎠ − Ki = 0 f i (ξ1 , . . . , ξR ) = ⎝ x j i ⎠ − Ki = ⎝ M R n + j =1 0j k=1 βk ξk j =1 j =1 1. Show that the (i, k)th element of the Jacobian of f is given by ⎛ ⎞ M  vij vkj ∂f i β β i k⎠ = (f i + Ki ) ⎝ − ∂ξk nj nT where n j = n 0j + on f i , i.e.,

R k=1

j =1

vkj ξk and n T =

M

j =1

n j . (Hint: Use the chain rule

 ∂f i ∂x j ∂f i = ∂ξk ∂x j ∂ξk M

j =1

and simplify further by using the definition of f i .) Next, show that the Jacobian can also be put in matrix form as d J = f = DS (2.82) dξ where D and S are a diagonal matrix and a symmetric matrix, respectively, given by ⎞ ⎛ 0 f 1 + K1 ⎟ ⎜ .. D=⎝ ⎠ . ⎛

f R + KR

0 ⎛

⎜ ⎜⎜ S=V⎜ ⎜⎝ ⎝

n −1 1 0

0 ..

. n −1 M

⎞

⎛

1 1 ⎜ . ⎟ ⎠− ⎝ .. nT 1

··· .. . ···

1 .. . 1

⎞ ⎞ ⎟ ⎟⎟ T ⎠⎟ ⎟V ⎠

which means that the Newton update is given by k ξ = −J −1 f = −  S−1 D−1 f ξ(k) . (Note: Because S involves n −1 j , one has to avoid choosing an initial value guess that would make any of the n j equal to zero.)

98

Solution of Multiple Equations

2. Consider the following equilibrium reactions: Reaction 1: A + B

⇔

C+D

Reaction 2: A + C

⇔

2E

with equilibrium constants, K1 = 0.4 and K2 = 4/15. Assume initial moles to be n 0,A = 100, n 0,B = 100, n 0,C = 0, n 0,D = 0 and n 0,E = 0. Use Newton’s method (i.e., write your own MATLAB m-file that implements the Jacobian formulas obtained in the previous question) to find the extents of reactions, ξ1 and ξ2 using as initial guesses: (0)

(0)

(0)

(0)

(0)

(0)

(a) Case 1: ξ1 = 50 and ξ2 = 20 (b) Case 2: ξ1 = 90 and ξ2 = 2 (c) Case 3: ξ1 = 90 and ξ2 = 11 If the solutions converge, evaluate the equilibrium composition. 3. Use the nonlinear solver nsolve.m that is available on the book’s webpage to find the solution with the initial guesses of preceding Cases 2 and 3. Implement both the line search (type 1) and the double-dogleg search (type 2). If the MATLAB Optimization toolbox is available, test these cases also using the built-in MATLAB function fsolve. E2.20. Use the Levenberg-Marquardt method (cf. B.14) to find the parameters α1 , . . . , α7 of the model y = α1 tanh (α2 x + α3 ) + α4 tanh (α5 x + α6 ) + α7 Table 2.9. Data for problem E2.20 x 0.7488 1.7857 3.0530 4.4355 5.0115 5.9332 7.5461 10.5415 13.4217

y 2.3816 2.3816 2.3728 2.4518 2.5658 2.7763 2.9956 3.0395 3.0570

x 15.2650 19.4124 21.2558 25.4032 27.1313 27.8226 29.3203 31.0484

y 3.0132 3.0044 2.9956 2.7061 2.4868 2.2939 1.9430 1.5658

x 31.8548 34.1590 35.6567 37.6152 40.3802 43.7212 47.6382 49.4816

y 1.4167 0.9956 0.8377 0.7412 0.6711 0.6360 0.6360 0.6447

that would yield a least-squares fit of the data given in Table 2.9. Also, a MATLAB file Levenmarg.m that implements the LevenbergMarquardt algorithm is available (Note: Use a random Gaussian number generator to obtain the initial guess).

3

Matrix Analysis

In Chapter 1, we started with the study of matrices based on their composition, structure, and basic mathematical operations such as addition, multiplication, inverses, determinants, and so forth, including matrix calculus operations. Then, in Chapter 2, we focused on the use of matrices to solve simultaneous equations of the form Ax = b, including their applications toward the solution of nonlinear equations via Newton algorithms. Based on linear algebra, we saw that Ax can also be taken to be a linear combination of the columns A with the elements of x acting as the weights. Under this perspective, the least-squares problem shifted the objective to be that of finding x that would minimize the residual error given by r = b − Ax. In this chapter, we return to the equation Ax = b with a third perspective. Here, we consider matrix A to be an operator that will “transform” (or “map”) an input vector x to yield an output vector b, as shown schematically in Figure 3.1. We call this the matrix operator perspective of the linear equation. The main focus is now on A as a “machine” that needs to be analyzed, constructed, or modified to achieve some desired operational characteristics. For instance, we may want to construct a matrix A that rotates, stretches, or flips various points xi described by vectors. As another example, a stress tensor (to be discussed in Chapter 4) can be represented by a matrix T , which can then be used to find the stress vector s pointing in the direction of a unit vector n by the operation s = T n. We begin with some general matrix operators in Section 3.1. These include unitary or orthogonal operators, projection operators, reflector operators, and so forth. We also include an affine extension of the various operators that would allow translation. Then, in Section 3.2, we introduce the important properties called eigenvalues and objects called eigenvectors, which characterize the behavior of the matrix as an operator. We also outline the properties of eigenvalues and eigenvectors of specific classes of matrices. As we show in later chapters, the eigenvalues are very important tool, for determining stability of differential equations or iterative processes. We also include an alternative numerical approach to the evaluation of eigenvalues and eigenvectors, known as the QR method. This method is an extension of another method known as the power method. Details of the QR method and the power method are included in Section C.2 as an appendix. Another application of eigenvalue analysis is also known as spectral analysis. It provides useful factorizations such as diagonalization, Jordan-block forms, singular value decompositions, and polar decompositions. In Section 3.7, we discuss 99

100

Matrix Analysis

Figure 3.1. Matrix as an operator.

an important result known as the Cayley-Hamilton theorem, which is very useful in evaluating analytical matrix functions. These tools provide valuable insights into functions of a matrix operator. As one application of spectral analysis, we briefly discuss the principal component analysis in Section 3.9.2. This type of analysis is used to reduce the dimensions of Linear models. Finally, in Section 3.11, we provide a brief discussion on matrix norms and condition numbers. These norms are important to sensitivity and error analysis of matrix operators. For instance, the condition numbers can help determine whether the solution of Ax = b will have a large error or not when small perturbations are introduced in A or b.

3.1 Matrix Operators Let A[=]N × M be a matrix that transforms an M-dimensional vector x to an Ndimensional vector b. This is represented by the same linear equation Ax = b. Generally, x does not reside is the same space as b, even when N = M. For instance, for the stress tensor equation T n = s, although n and s are both vectors of the same length, s has units of stress, that is, force per unit area, whereas n does not. Matrix operators are also known as linear mappings because they obey the following property: A (αx + βy) = αAx + βAy

(3.1)

where α and β are scalars. The associative properties of matrices can then be viewed as compositions of operator sequences as follows: A (Bx) = b

→

(AB) x = b

This is shown in Figure 3.2. As we saw in Chapter 1, matrix products are not commutative in general. This means that when C = AB operates on x, operator B is applied first to yield y = Bx. Then y is operated on by A to yield b.

Figure 3.2. Composition of operators.

3.1 Matrix Operators

101

When using the same operator on several input vectors, one could also collect the input vectors into a matrix, say X, and obtain a corresponding matrix of output vectors B.     AX = A x1 x2 · · · xP = b1 b2 · · · bP = B Note that although X and B are matrices, they are not treated as operators. This is to emphasize that, as always, the applications will dictate whether a matrix is viewed an operator or not.

3.1.1 Orthogonal and Unitary Matrix Operators Definition 3.1. A square matrix A is an unitary matrix if A∗ A = AA∗ = I

(3.2)

A square matrix A is an orthogonal matrix if AT A = AAT = I

(3.3)

Unitary matrix operators are operators that preserve Euclidean norms defined as

# $ n $ √ x = % xi xi = x∗ x i=1

To see this, b2 = Ax2 = (Ax)∗ (Ax) = x∗ (A∗ A)x = x∗ x = x2 When the matrices and vectors are real, unitary operators are synonymous with orthogonal operators. Thus we also get for real vectors and matrices, b2 = Ax2 = (Ax)T (Ax) = xT (AT A)x = xT x = x2 Note that if A is a unitary (or orthogonal) operator, the inverse operator is found by simply taking the conjugate transpose (or transpose), that is, A−1 = A∗ (or A−1 = AT ). In the following examples, unless explicitly stated, the matrix operators and vectors are understood to have real elements. Examples of orthogonal matrices include the following: 1. Permutation operators. These matrices are obtained by permuting the rows of an identity matrix. When operating on an input vector, the result is a vector with the coordinates switched according to sequence in the permutation P. For instance, ⎛ ⎞ ⎛ ⎞⎛ ⎞ ⎛ ⎞ x 0 0 1 x z P⎝ y ⎠ = ⎝ 1 0 0 ⎠⎝ y ⎠ = ⎝ x ⎠ z 0 1 0 z y 2. Rotation operators (also known as Givens operator). The canonical rotation operators R are obtained from an identity matrix in which two elements in the diagonal, say, at rk,k and r, , are replaced by cos (θ), and two other off-diagonal

102

Matrix Analysis 2

1.5 x

1

x2, b2

0.5

0

b=R

cw

−0.5

x

Figure 3.3. Rcw (θ) rotates x clockwise by θ radians.

−1

−1.5

−2

−2

−1

0

1

2

x ,b 1

1

elements at rk, and r,k are replaced by sin (θ) and − sin (θ), respectively. For a two-dimensional rotation, we have   cos(θ) sin(θ) (3.4) Rcw(θ) = − sin(θ) cos(θ) as a clockwise rotation of input vectors. To see that Rcw is indeed an orthogonal matrix, we have    cos(θ) sin(θ) cos(θ) − sin(θ) T Rcw Rcw = − sin(θ) cos(θ) sin(θ) cos(θ)   0 cos2 (θ) + sin2 (θ) =I = 0 cos2 (θ) + sin2 (θ) Similarly, we can show that Rcw RTcw = I. To illustrate the effect of clockwise rotation, consider  √    1 2 x= → b = Rcw(π/4) x = 1 0 The original point was rotated π/4 radians clockwise, as shown in Figure 3.3. Because Rcw is orthogonal, the inverse, operator, is the counterclockwise rotation obtained by simply taking the transpose of Rcw , that is,   cos(θ) − sin(θ) −1 Rccw(θ) = Rcw (θ) = (3.5) sin(θ) cos(θ) For three dimensions, the clockwise rotation operators around x, y, and z, assuming the vectors are arranged in the classical order of (x, y, z)T , are given by Rcw,x , Rcw,y , and Rcw,z, respectively, ⎛ ⎛ ⎞ ⎞ 1 0 0 cos(θ) 0 − sin(θ) ⎠ Rcw,x = ⎝ 0 cos(θ) sin(θ) ⎠ Rcw,y = ⎝ 0 1 0 0 − sin(θ) cos(θ) sin(θ) 0 cos(θ) ⎛ ⎞ cos(θ) sin(θ) 0 Rcw,z = ⎝ − sin(θ) cos(θ) 0 ⎠ 0 0 1 and shown in Figure 3.4.

3.1 Matrix Operators z

z

z

y

y

y

x

x

R

103

x

R

R

Figure 3.4. 3D clockwise rotation operators.

3. Reflection operators (also known as Household operators). These operators have the effect of reflecting a given point symmetrically along a given hyperplane. The hyperplane is determined by the direction of a nonzero vector w that is normal to the plane. The Householder transformation operator, Hw , based on w, is defined as Hw = I −

2 ww∗ w∗ w

(3.6)

Note that Hw is also Hermitian, that is, Hw∗ = Hw . The action of Hw on a vector x is to reflect it with respect to a hyperplane that is perpendicular to w. This is shown in Figure 3.5. To show that Hw is unitary, we have Hw∗ Hw = Hw Hw∗ = Hw2

= = =

2 2 ww∗ )(I − ∗ ww∗ ) w∗ w w w 4 4 I − ∗ ww∗ + ∗ 2 ww∗ ww∗ w w (w w) (I −

I−

4 4 ww∗ + ∗ ww∗ = I w∗ w (w w)

The Householder operator could also be used as an alternative way to move a particular vector into another direction specified by a vector (instead of specifying angles). For instance, let x be the input and y the desired output, where both have the same norm but pointing in different directions. By setting w = y − x, H(y−x) will move x to have the same direction as y. To show this, use the fact that

Hw x

Figure 3.5. Householder transformation as a reflector operator.

origin x w

104

Matrix Analysis

x∗ x = y∗ y and x∗ y = y∗ x. Then,  H(x−y) x = I−

2 (x − y)∗ (x − y)

(x − y) (x − y)

∗

 x

1 (xx∗ x − yx∗ x − xy∗ x + yy∗ x) − y∗ x

=

x−

=

xx∗ x − xy∗ x − xx∗ x + yx∗ x + xy∗ x − yy∗ x x∗ x − y∗ x

=

y

x∗ x

(3.7)

3.1.2 Projection Operators Projection operators are another important class of operators. Generally, they separate a given space into two complementary subspaces: one subspace is the region where the outputs of P will reside, and the other subspace is specified by its complement, (I − P). Definition 3.2. A matrix operator P is idempotent if and only if P2 = P. If in addition, P is hermitian, then it is known as a projection operator. If P is a projection operator, then (I − P) is known as the complementary projection operator. We can see that if P is idempotent, then so is its complement, (I − P), that is, (I − P)2

=

(I − P) (I − P)

=

I − 2P + P2

=

I−P

Let v reside in the linear space L, and let S be a subspace in L. Then a projection operation, PS , can be used to decompose a vector into two complementary vectors, v

=

(PS + I − PS ) v

=

Pv + (I − P) v

=

vS + vL−S

where vS will reside in the subspace S, whereas vL−S will reside in the complementary subspace) L − S. * Let q1 , . . . , qM be a set of orthonormal basis vectors for S, that is, q∗i q j

+ =

0 1

if i = j if i = j

* ) S = Span q1 , . . . , qM

and

Then a projection operator onto S is given by PS =

M  i=1

qi q∗i

(3.8)

3.1 Matrix Operators

105

This can be generalized to a set of Linearly independent vectors {a1 , . . . , aM }, with S = Span {a1 , . . . , aM }. One can use the QR algorithm (cf. Section 2.6) to obtain  and  matrix Q R where  Q R=A

and

∗Q =I Q

 then these columns will be a set of orthonormal Let Q be the first M columns of Q; basis vectors for S. Also, let R be the first M rows of  R. One can check that QR = A. The projection operator is then given by PS = QQ∗

(3.9)

with the following properties: 1. PS is Hermitian. 2. For any vector v, PS v will be in the span S. To show this, QQ∗ v =

M  i=1

qi (q∗i v) =

M 

αi qi ∈ S

i=1

where αi = q∗i v. 3. For any vector v and b, y = PS v is orthogonal to z = PL−S b, where PL−S = I − PS . To show this, y∗ z = v∗ PS∗ PL−S b = v∗ QQ∗ (I − QQ∗ ) b = v∗ (QQ∗ − QQ∗ QQ∗ ) b = 0 There is also a strong relationship between the projection operator and the least-squares solution of Ax = b given by (2.35). Using the QR factorization of A, =

(A∗ A)−1 A∗ b = (R∗ Q∗ QR)−1 R∗ Q∗ b

=

R−1 Q∗ b

R x

=

Q∗ b

QR x

=

QQ∗ b

A x

=

PS b

 x

Thus A x is the orthogonal projection of b onto the subspace spanned by the columns of A.

EXAMPLE 3.1. Suppose we want to obtain the projection operator of a threedimensional vector b onto the subspace spanned by a1 and a2 , where ⎛ ⎛ ⎛ ⎞ ⎞ ⎞ 0.5 1 0 b = ⎝ 0.5 ⎠ a1 = ⎝ 0 ⎠ a2 = ⎝ 1 ⎠ −1 −1 −2   Let A = a1 a2 . Using the QR-algorithm on A, we obtain the orthonormal basis, √ ⎛ √ ⎞ 2/2 − √3/3   ⎠ Q = q1 q2 = ⎝ √ 0 √3/3 − 2/2 − 3/3

106

Matrix Analysis

1.4 A( line

(a,b)

1.2

1

)

line(a,b)

A( line

(c,d)

)

0.8

y

0.6

Figure 3.6. Using a linear operator A to translate line(a,b) may not translate another line line(c,d) .

line

(c,d)

0.4

0.2

0

−0.2 0

0.5

1

1.5

2

x

then the projection operators are given by ⎛ ⎞ ⎛ 5 −2 −1 1 1⎝ 1⎝ ∗ ⎠ PS = QQ = −2 2 −2 P(L−S) = I − PS = 2 6 6 −1 −2 5 1 Solving for the least-squares solution, ⎛ ⎞ ⎛ ⎞ 1 0 0.5 ⎝ 0 1 ⎠ x = ⎝ 0.5 ⎠ → −1 −2 −1

 x = R−1 Q∗ b =

1 12



5 4

2 4 2

⎞ 1 2 ⎠ 1



We can check that A x = PS b.

3.1.3 Affine Operators A linear operator can often be constructed to translate one line at a time. However, for a given set of nonparallel lines, it is not possible to translate all these lines in the same fashion. To illustrate, consider the line segments line(a,b) and line(c,d) , where         1 0 0 1 a= b= c= d= 0 1 0.5 0.5 A matrix operator A given by  A=

1.75 0.25

0.75 1.25



will translate line(a,b) but not line(c,d) , as shown in Figure 3.6. A generalization of linear operators that include translations is called affine operators. Definition 3.3. An affine operator acting on vector v , denoted by Aff (A,t) (v), is a linear operator A acting on v followed by a translation t, that is, Aff (A,t) (v) = Av + t

(3.10)

3.2 Eigenvalues and Eigenvectors

107

Strictly speaking, affine operators are not linear operators. However, we can transform the affine operation and make it a linear matrix operation. One approach is to expand both the operator A and the input vector v as follows:     A t v   A= v= (3.11) 0 1 1 where vector t is the desired translation. After the affine operation  A has finished operation on  v, one can then obtain the desired results by simply removing the last element of  v, that is,          A t   Av + t v A v= I 0 I 0  = I 0 = Av + t 0 1 1 1 T  T Given a line segment defined by its endpoints, a = ax , ay and  b = bx , by . Suppose we want to rotate T this line segment counterclockwise , m , where mx = (ax + bx ) /2 and my = by θ radians at its midpoint, m = m x y   ay + by /2. This operation can be achieved by a sequence of affine operations. First, we can translate the midpoint of the line to the origin. Second, we rotate the line. Lastly, we translate the origin back to m. In the 2-D case, this is given by ⎛ ⎞⎛ ⎞⎛ ⎞ 1 0 mx cos (θ) − sin (θ) 0 1 0 −mx cos (θ) 0 ⎠ ⎝ 0 1 −my ⎠ Aff = ⎝ 0 1 my ⎠ ⎝ sin (θ) 0 0 1 0 0 1 0 0 1 ⎛ ⎞ cos (θ) − sin (θ) α cos (θ) β ⎠ = ⎝ sin (θ) 0 0 1

EXAMPLE 3.2.

where α

=

(1 − cos (θ)) mx + sin (θ) my

β

= − sin (θ) mx + (1 − cos (θ)) my  T  T To illustrate, for a = 0.5 0.5 , b = 0.7 0.5 and θ = π/4, the affine operation is shown in Figure 3.7.

3.2 Eigenvalues and Eigenvectors To study the characteristics of a particular matrix operator A, one can collect several pairs of x and b = Ax. Some of these pairs will behave more distinctively than others, and they yield more information about the operator. A group of vectors known as eigenvectors have the distinct property such that the only effect that A has on them is a scaling operation. Definition 3.4. Let A[=]N × N, then v = 0 is an eigenvector of A if Av = λv where λ is a scalar called the eigenvalue.

(3.12)

108

Matrix Analysis

0.8

0.7

y Aff( line

0.6

(a,b)

line(a,b)

0.5

0.4

Figure 3.7. Applying affine operation Aff that rotates a line segment line(a,b) counterclockwise by θ = π/4 radians around its midpoint.

)

0.4

0.5

0.6

0.7

0.8

x Because v = 0 is always a solution to (3.12), it does not give any particular information about an operator. It is known as the trivial vector. This is why we only consider nonzero vectors to be eigenvectors. To evaluate the eigenvectors of A, we can use the condition given by (3.12). Av = λv

→

(A − λI) v = 0

(3.13)

For v = 0, we need (λI − A) to be singular, that is, det ( λI − A ) = 0

(3.14)

Equation (3.14) is known as the characteristic equation of A, and this equation can be expanded into a polynomial of order N, where N is the size of A. Using the formula for determinants given by (1.10), we have the following lemma: Let A[=]N × N. Then the characteristic equation (3.14) will yield a polynomial equation given by

LEMMA 3.1.

charpoly(λ) = λN + βN−1 λN−1 + · · · + β1 λ + β0 = 0 where βN−k = (−1)k

 1

|aki |

i=1,i=k

Using the result about Gershogorin circles given in the previous exercise, show that a diagonally dominant matrix is nonsingular. E3.22. Let v be an eigenvector matrix A. Show that the corresponding eigenvalue for v can be obtained by using the Rayleigh quotient defined as v∗ Av (3.75) v∗ v E3.23. Use the power method (cf. Section C.2.2) to find the dominant eigenvalue of ⎛ ⎞ 4 2 3 A = ⎝ −3 0 2 ⎠ 1 1 4 λ=

E3.24. Let A have distinct eigenvalues; any vector w can be represented as a linear combination of the eigenvectors, that is, w=

N 

αi vi

i=1

where vi is the ith eigenvector of A. Assume that |λ1 | ≥ |λk |, k > 1 Then, Aw

=

N  i=1

=

αi λi vi ⎛

α1 λ1 ⎝v1 +

N  j =2

⎞ λj αj vj ⎠ λ1

Multiply this equation by Ak and use Property 5 of Section 3.3 to show that the power method will approach α1 λk+1 v1 (or V1 if normalization is performed) as k → ∞. This is the basis for the power method. E3.25. Consider the data given in Table 3.3 and plotted in Figure 3.15. We want to use principal component analysis to obtain a 2D curve in 3D space. 1. Obtain the mean-adjusted data matrix  A and take the reduced singular   value decomposition of A, that is, A = UV ∗ 2. Obtain the projection matrix PV that would project the mean-adjusted data onto the space spanned by the first two columns of V . (This assumes that the last column is attached to the smallest singular value).

144

Matrix Analysis Table 3.3. Raw data set for the principal component analysis x

y

z

x

y

z

x

y

z

0.0104 0.0288 0.0311 0.0449 0.0657 0.0703 0.1048

0.0541 0.1944 0.2558 0.3085 0.3816 0.4371 0.4898

1.3846 1.0956 0.7672 0.6511 0.3595 0.1628 0.1700

0.1302 0.2131 0.2615 0.3145 0.4090 0.4666 0.5288

0.5161 0.5482 0.5629 0.6096 0.6096 0.5629 0.5512

0.2402 0.2274 0.2011 0.4074 0.4178 0.7435 0.9040

0.6302 0.7523 0.7915 0.9021 0.9389 0.9597 0.9873

0.5395 0.5395 0.5892 0.6213 0.7295 0.7646 0.9342

1.1170 1.4258 1.2890 1.2737 1.0864 1.1498 0.5464

1.5

z 1

1

0.5

0 1

0.5

Figure 3.15. Plot of raw data for the principal component analysis.

x

0.5

y

0

0

3. Using both P and V , we end up with a 2D surface in which we could perform additional regression analysis. Thus let B be the transformed data matrix   A (PV ) B=  The last column of B should be all zeros. Let xt and yt represent the first and second columns of B, respectively, of the transformed data. 4. A plot of xt vs. yt is given in Figure 3.16. Note that the curve can not be described by yt = yt (xt ). However, we can rotate the curve 60◦ 1

0.5

yt

0

Figure 3.16. Plot of data transformed into 2D space.

−0.5

−1 −1

−0.5

0

xt

0.5

1

3.12 Exercises

145

counter-clockwise using Rccw(60◦ ) . This will then make it possible for the nonlinear regression to be applied to the rotated curve. Let   C = B•,1 B•,2 RTccw(60◦ ) and let x f and y f represent the first and second columns of C. Obtain a fifth-order polynomial regression model for the rotated curve, that is, y f = a0 + a1 x f + a2 x2f + a3 x3f + a4 x4f + a5 x5f 5. By reversing the operations, we can transform the regression curve y f = y f (x f ) back to be in terms of the original variables. Thus perform these operations on the regression curve to the curve shown in Figure 3.17. 1.5

z 1 Figure 3.17. Plot of the data together with the regression curve in the original space.

0.5

0 1

2

1 0.5

y

0 0 −1

E3.26. Find the least-squares solution of the following equation using the SVD decomposition using a tolerance level of 10−4 (i.e., set J k = 0 if J k ≤ 10−4 ) and the reduced SVD decomposition: ⎛ ⎞ ⎞ ⎛ 0.7922 0.6555 0.1367 2.5583 ⎜ 3.7049 ⎟ ⎜ 0.9595 0.1712 0.7883 ⎟ ⎜ ⎟ ⎟ ⎜ ⎜ 1.9351 ⎟ ⎟ ⎜ 0.6557 0.7060 −0.0503 ⎟ ⎜ ⎟ ⎜ ⎜ 0.1488 ⎟ ⎜ 0.0357 0.0318 0.0039 ⎟ ⎜ ⎟ ⎟ ⎜ ⎜ ⎟ ⎜ 0.8491 0.2769 0.5722 ⎟ ⎟ x = ⎜ 3.1600 ⎟ ⎜ ⎟ ⎟ ⎜ 0.9340 ⎜ 0.0462 0.8878 ⎟ ⎜ ⎜ 3.7059 ⎟ ⎟ ⎟ ⎜ 0.6787 ⎜ 0.0971 0.5816 ⎟ ⎜ ⎜ 2.6256 ⎟ ⎜ 0.7577 ⎜ 2.2373 ⎟ 0.8235 −0.0657 ⎟ ⎟ ⎟ ⎜ ⎜ ⎠ ⎝ 0.7431 ⎝ 2.3301 ⎠ 0.6948 0.0483 0.3922 0.3171 0.0751 1.2753 E3.27. With the Frobenius norm of A[=]N × M defined as # $ N M $   2 aij  AF = % i=1 j =1

Show that another method can be used to evaluate AF , for example, " AF = trace (A∗ A)

x

PART II

VECTORS AND TENSORS

The next two chapters contain a detailed discussion of vector and tensor analysis. Chapter 4 contains the basic concepts of vectors and tensors, including vector and tensor algebra. We begin with a description of vectors as an abstract object having a magnitude and direction, whereas tensors are then defined as operators on vectors. Several algebraic operations are summarized together with their matrix representations. Differential calculus of vector and tensors are then introduced with the aid of gradient operators, resulting in operations such as gradients, divergences, and curls. Next, we discuss the transformations of rectangular coordinates to curvilinear coordinates, such as cylindrical, spherical, and other general orthogonal coordinate systems. Chapter 5 then focuses on the integral calculus of vectors. Detailed discussions of line, surface, and volume integrations are included in the appendix, including the mechanics of calculations. Instead, the chapter discusses various important integral theorems such as the divergence theorem, the Stokes’ theorem, and the general Liebnitz formula. An application section is included to show how several physical models, especially those based on conservation laws, can be cast in terms of tensor calculus, which is independent of coordinate systems. The models generated are generally in the form of partial differential equations that are applicable to problems in mechanics, fluid dynamics, general physico-chemical processes, and electromagnetics. The solutions of these models are the subject of Part III and Part IV of the book.

147

4

Vector and Tensor Algebra and Calculus

In this chapter, we work with objects that possess a magnitude and a direction. These objects are known as physical vectors or simply vectors.1 There are two types of vectors: bound vectors, which are fixed to a specified point in the space, and free vectors, which are allowed to move around in the space. Ironically, free vectors are often used when working in rigid domains, whereas bound vectors are often used when working in flowing or flexible domains. We mostly deal with bound vectors. We denote vectors with underlined bold letters, such as v, and we denote scalars with nonunderlined letters, such as α, unless otherwise noted. Familiar examples of vectors are velocity, acceleration, and forces. For these vectors, the concept of direction and magnitude are natural and easy to grasp. However, it is important to note that vectors can be built depending on the user’s interpretation and objectives. As long as a magnitude and direction can be attached to a physical property, then vector analysis can be used. For instance, for angular velocities of a rigid body, one needs to describe how fast the rotation is, whether the rotation is counterclockwise or clockwise, and where the axis of rotation is. By attaching an arrow whose direction is along the axis of rotation, whose length determines how fast the rotation is, and pointing in the direction consistent with a counterclockwise or clockwise convention, the angular velocity becomes a vector. In our case, we adapt the right-hand screw convention to represent the counterclockwise direction as a positive direction (see Figure 4.1). We begin in Section 4.1 with the description of fundamental vector operations. The definitions of operations such as vector sums and different type of products, including scalar, dot, cross, and triple, are done in a geometric sense, that is, based only on measurements of distance and angles. Later, we introduce unit basis vectors such as δx , δy , and δz in the rectangular coordinates pointing in the x, y, and z directions, respectively. When vectors are represented as linear combinations of the basis unit vectors, an alternative set of efficient calculations can be achieved. These basis vectors are also used to define an operator known as a tensor. Moreover, vectors and tensors can also be represented by matrices. In doing so, we can take advantage of various matrix properties and apply matrix analysis and computation 1

Physical vectors can also be represented by matrix vectors. We defer this matrix representation until Section 4.3.

149

150

Vector and Tensor Algebra and Calculus

Figure 4.1. The right-hand screw convention.

to handle vector operations; that is, the concepts of eigenvalues, eigenvectors, polar decomposition, diagonalization, and so forth can be applied to the physical vectors and tensors. In Section 4.5, we discuss the derivatives of vectors that are dependent on one variable. The derivative of such a vector is different from that of a scalar function because vectors have the additional property of directions. Furthermore, the results of the derivatives are also vectors. The derivative of sums and different types of products can also be obtained. Due to its ubiquitous importance, we briefly apply the derivatives to position vectors r and its various derivatives, for example, the velocity vector and acceleration vector, including other related items such as tangent vectors, normal vectors, binormal vectors, curvature, and torsion. Then, in Section 4.7, we discuss the differential calculus of vector fields, where a distribution of vectors are specified at different locations of the 3D space. This includes differential operations such as the gradient, divergence, curl, and Laplacian, among others. Finally, in Sections 4.8 and 4.9, we discuss alternative coordinate systems, namely cylindrical, spherical, and general orthogonal coordinate systems. These are important coordinate systems to consider because in several real-world applications, the boundaries are cylindrical (e.g., pipe flow) or spherical (e.g., heat transfer from a spherical surface). Starting with the basic transformation rules between the coordinates, we can generate relationships with those in the rectangular coordinate systems. Unfortunately, for differential operators, changing the representations based on the other coordinate systems tends to be more complicated. Nonetheless, the formulas for gradient, divergence, curl, and Laplacian can still be generated in a straightforward manner.

4.1 Notations and Fundamental Operations In this section, we describe the fundamental vector operations. These are summarized in Table 4.1. The geometric interpretations are shown in these tables. However, one should be careful when applying these interpretations on vectors that have physical units. We briefly explore some of these concerns. Because most vectors have physical units, we attach all the units with the “magnitude” component of the vector. Thus, by defining the norm, v , as the magnitude of the v, we can represent v as v = v nv

where

nv =

1 v v

(4.1)

Here, nv is the “normalized” unit vector of v, and it will not have any physical units attached to it.

4.1 Notations and Fundamental Operations Table 4.1. Fundamental vector operations Operation

Notation

Procedure

1

Addition

c=a+b

Let a and b be attached to point O. Construct a parallelogram with a and b as two adjacent sides. Then c is the vector at point O with its arrowhead located opposite point O in the parallelogram.

2

Norm

α= v

α is the length of v, from the tail to the arrowhead.

3

Scalar Product

w = αv

Direction of w = direction of v; w = α v

4

Dot Product

α=u·v

α = u v cos (θ) where, θ = smaller angle between u and v.

5

Cross Product

c=a×b

Direction of c = perpendicular to plane containing a and b, based on the right-hand screw convention. c = a b sin θ = area of parallelogram formed by a and b. where, θ = angle from a to b based on the right-hand screw convention

6

Triple Product

α = c · (a × b) = [c a b]

α = volume of parallelepiped formed by a, b, and c. α = a b c sin θ cos φ where, θ = angle from a to b based on right-hand screw convention, and φ = angle between c and a line that is perpendicular to the plane containing a and b.

151

152

Vector and Tensor Algebra and Calculus

Let c be the sum of vectors a and b; then c is also called the resultant, and c should be in the plane containing both a and b. Furthermore, a, b, and c should all have the same physical units. This is necessary for the following cosine law to apply, c

2

= a

2

+ b

2

−2 a

b cos (π − θ)

(4.2)

where θ is the angle between a and b. Also, the angle between b and c is given by  −1

γ = sin

a c

 sin(θ)

(4.3)

There are four types of vector products: scalar, dot, cross, and triple products. The resulting products will most likely not be the same as factored vectors, both in units or in meaning. Thus care must be taken when plotting different type of vectors in the same 3D space. The properties for the vector operations are given in Table 4.2. The table is grouped based on the operations involved (i.e., sums, scalar products, dot products, cross products, triple products, and norms). Most of the properties for the sums, scalar products, and dot products are similar to the vectors of matrix theory. On the other hand, the properties of cross products are quite different. First, we see that cross products are anti-commutative; that is, the sign is reversed if the order of the vectors are interchanged. Next, the parallelism property states that two nonzero vectors are parallel to each other if their cross product is the zero vector.2 Finally, cross products are not associative. This means that the use of parentheses is imperative; otherwise, the operation will be ambiguous. The properties of both the sums and scalar products show that the space of physical vectors satisfies the conditions given in Table B.3 for a linear vector space. This means that the properties and definitions attached with linear vector spaces are also applicable to the space of physical vectors. These include the definitions of linear combination, linear independence, span, and dimension, as given in Table B.2.3 Thus ) * 1. A set of vectors, v1 , . . . , vn are linearly independent if the only possible linear combination that results in a zero vector, α1 v1 + · · · + αn vn = 0 is when α1 = · · · = αn = 0. ) * 2. A set of linearly independent vectors, V = v1 , . . . , vn is a basis for a space S, if S is the span of the vectors of V . 3. The dimension of a subspace S is the number of linearly independent vectors that would span S.

2

3

This is a dual property to orthogonality, in which two nonzero vectors are orthogonal if their dot product is zero. However, one should note that the parallel property results in a vector, whereas the orthogonality property results in a scalar. In fact, the abstract linear space is the generalization of the space of physical vectors.

4.1 Notations and Fundamental Operations Table 4.2. Linear properties of physical vectors Vector Sums   v + w + y = (v + w) + y

1

Associative

2 3 4

Commutative Identity is 0 Inverse exist and unique

v+w=w+v 0+v=v v + (−v) = 0

Scalar Products 1 2 3 4

α (βv) = (αβ) v 1v = v (α + β) v = αv + βv

Associative Identity is 1 Vector is distributive over scalar sums Scalar is distributive over vector sums

α (v + w) = αv + αw Dot Products

1 2 3 4 5

2

v =v·v v·w=w·v u · (v + w) = u · v + u · w α (v · w) = (αv) · w = v · (αw) u·v=0 if u = 0, v = 0 or u⊥v

Squared norm Commutative Distributive Scalar Product Orthogonality Cross Products

1 2 3 4

Anti-commutative Distributive Scalar Product Parallelism

5

Non-associativity

1

Cyclic Permutation

v × w = −w × v u × (v + w) = u × v + u × w α (v × w) = (αv) × w = v × (αw) u×v=0 if u = 0, v    =0 or u || v

u × v × y = u · y v − (u · v) y     (u × v) × y = y · u v − y · v u Triple Products u · (v × w) = w · (u × v) = v · (w × u) Norms

1 2 3 4

Positivity Scaling Triangle Inequality Unique Zero

v ≥ 0 αv = |α| v v+w ≤ v + w v = 0 only if v = 0

EXAMPLE 4.1. Consider the set of 3D vectors a, b, c, and d and a 2D plane S as shown in Figure 4.2. Suppose further that b, c, and d lie in the plane S. Then, r The only sets of three linearly independent vectors are: )a, b, c*, )a, b, d*, ) * a, c, d .) r The set b, c, d* is linearly dependent. r All the sets of any two different vectors are linearly independent, for example, ) * ) * a, b , a, c ,)and*so forth. r The span of b, c is S, and thus the dimension of S is 2. Also, the spans of ) * ) * ) * b, c , c, d , and b, d are all the same.

153

154

Vector and Tensor Algebra and Calculus

Figure 4.2. A set of 3D vectors for Example 4.1.

4.2 Vector Algebra Based on Orthonormal Basis Vectors In most applications, vectors can be represented as a linear combination of basis vectors. For instance, if b1 , b2 , and b3 are three linearly independent 3D vectors, then any 3D vector v can be represented as v = αb1 + βb2 + γb3

(4.4)

The scalars α, β, and γ are known as the components along b1 , b2 , and b3 vectors, respectively. In particular, it the basis vectors are normalized (magnitude 1 and no physical units) and orthogonal to each other, the set of basis unit vectors are known as orthonormal basis vectors. We start with orthonormal basis vectors based on the Cartesian coordinate system, also known as the rectangular coordinate systems, described by (x, y, z), using the convention shown in Figure 4.3.4 The unit vectors based on the Cartesian coordinates are denoted by δx , δy , and δz, each pointing in the positive x, y, and z direction, respectively. Thus v = vx δx + vy δy + vzδz

(4.5)

and the scalars vx , vy , and vz will be the x- , y-, and z- components of v, respectively (see Figure 4.4). The dot products and cross products of Cartesian unit vectors can be summarized as follows: δi · δ j

=

δij

(4.6)

δi × δ j

=

ijk δk

(4.7)

where δij is known as the Kronecker delta defined by + δij =

4

0 1

if i = j if i = j

(4.8)

The convention is that when looking down against the positive z-axis, the (x, y) plane, when rotated, should have the positive y-axis pointing vertically upward and the positive x-axis pointing horizontally to the right.

4.2 Vector Algebra Based on Orthonormal Basis Vectors

155

Figure 4.3. The Cartesian coordinate system with three axes perpendicular to each other. The figure on the right (a) is the relationship among x, y, and z. The figure on the left (b) is the convention by looking directly down into the positive z-direction.

and ijk is the permutation symbol (also known as Levi-Civita symbol) defined by

ijk

⎧ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨

0

1 ⎪ ⎪ ⎪ ⎪ ⎪ ⎩−1

if i = j or j = k or i = k if (i, j, k) = (x, y, z) or (i, j, k) = (z, x, y) or (i, j, k) = (y, z, x) (4.9) if (i, j, k) = (x, z, y) or (i, j, k) = (z, y, x) or (i, j, k) = (y, x, z)

Using (4.6) and the distributive property of dot products, the x-component of v can be found as follows: v · δx

=

  vx δx + vy δy + vzδz · δx

=

vx δx · δx + vy δy · δx + vzδz · δx = vx

Similarly, vy = v · δy and vz = v · δz. The following identities between the permutation symbols and Kronecker deltas will also be useful during the derivation of vector operations and properties: ⎛ ijk mn 

ijk imn

= =

δi

δim

δin

⎞

⎜ det ⎝ δ j

δ jm

⎟ δ jn ⎠

δk

δkm

δkn

δ jm δkn − δ jn δkm

(4.10)

i

Equations (4.6) and (4.7), together with the properties given in Table 4.2 yield alternative approaches to the operations defined in Table 4.1. These are given in Table 4.3.

Figure 4.4. Vector V as a linear combination of unit vectors.

156

Vector and Tensor Algebra and Calculus Table 4.3. Vector operations based on unit vectors Vector Addition

v+w

= =

  (vx + wx ) δx + vy + wy δy + (vz + wz) δz 

(vi + wi ) δi

i=x,y,z

Norm

v

= =

" v2x + v2y + v2z # $ $  2 % vi i=x,y,z

Scalar Product

αv

= =

  (αvx ) δx + αvy δy + (αvz) δz 

αvi δi

i=x,y,z

Dot Product

v·w

= =

  (vx wx ) + vy wy + (vzwz) 

δij vi w j =

i,j =x,y,z

Cross Product

v×w

=

=



vi wi

i=x,y,z

  vy wz − vzwy δx + (vzwx − vx wz) δy   + vx wy − vy wx δz  ijk vi w j δk i,j,k=x,y,z

Triple Product

u · (v × w)

=

=

  vy wz − vzwy ux + (vzwx − vx wz) uy   + vx wy − vy wx uz  ijk ui v j wk i,j,k=x,y,z

  Let us prove the identity for u × v × y given in Table 4.2. First, we can expand the cross products in terms of the permutation symbols: ⎛ ⎞ ⎛ ⎞     u× v×y = ⎝ ui δi ⎠ × ⎝  jk v j yk δ ⎠ EXAMPLE 4.2.

i=x,y,z

=

j,k,=x,y,z



im  jk ui v j yk δm

i,j,k,,m=x,y,z

Due to the cyclic nature of the symbols, we see that im = mi and  jk = jk . Thus, using (4.10),    u× v×y = mi jk ui v j yk δm i,j,k,,m=x,y,z

=



(δmj δik − δmk δij ) ui v j yk δm

i,j,k,m=x,y,z

=



i,j =x,y,z

=

(ui yi ) v j δ j −

  u · y v − (u · v) y

 i,k=x,y,z

(ui vi ) yk δk

4.3 Tensor Algebra

157

Some useful mnemonics exist for the cross product and the triple product. These are given by ⎛

v×w

=

u · (v × w)

=

δx ⎝ det vx wx ⎛ ux det ⎝ vx wx

δy vy wy uy vy wy

⎞ δz vz ⎠ wz ⎞ uz vz ⎠ wz

(4.11)

(4.12)

Note, however, that (4.11) and (4.12) are just memory tools and should not be treated as definitions. The unit basis vectors are normalized vectors, and thus they will have no physical units. The physical units are instead attached to the x-, y-, and z-components of the vector. Moreover, because scalars can move freely across the various types of products, the products of unit vectors can be computed in pure geometric terms. For instance, to compute for work we now have W = F · s

=



   f x δx + f y δy + f zδz · sx δx + sy δy + szδz

=

( f x δx ) · ( sx δx ) + · · ·

=

( f x sx ) (δx · δx ) + · · ·

where the products f x sx , and so forth are scalars with physical units of force times length, whereas δx · δx , and so forth are either 1 or 0 and do not have any units.

4.3 Tensor Algebra In this section, we discuss vector operators known as tensors. We begin with the definition of the building blocks for tensors, which are operators on vectors known as dyads. Definition 4.1. The dyad based on vectors v and w, denoted by (vw), is an operator that transforms an input vector x into output vector y as follows:   (vw) x

=

v (w · x)

=

αv = y

(4.13)

where α = w · x. One important concept in transport of properties such as material, energy, momentum, and so forth is the idea of flux, f . By flux, we mean the amount of property passing perpendicularly through a specified region per unit area of that region per unit time. This is a vector because the definition has both a magnitude and direction, although one has to be cautious about which

EXAMPLE 4.3.

158

Vector and Tensor Algebra and Calculus P

n

c

z

Figure 4.5. The triangle abc together with its normal unit vector.

a

y

b x

perpendicular direction is of interest; for example, for a closed surface, the unit normal vector to the surface can be chosen to be the outward normal.5 Consider the triangle abc determined by three non-collinear points a, b, and c, each defined by the position vectors a, b, and c, respectively, and shown in Figure 4.5.6 Assume that the sequence {a, b, c} yields a counterclockwise turn to yield our desired direction. Then the desired unit normal vector can be found to be (b − a) × (c − a) = n (b − a) × (c − a) Assume a constant property flow, P (e.g., kg of water per ft2 of region perpen. dicular to its flow), which does not necessarily flow in the same direction as n The magnitude of the flux through abc will need to take the projection of P  using the inner product. Thus the flux of P through triangle abc is given along n by    n  f = P·n or, in terms of the dyad notation,

  n  P f=n

Let P˙ abc be the total rate of property flowing through the triangle. Then P˙ abc can be obtained by taking the normal component of the flux and multiplying it by the area. The area of the triangle abc is given by A= Thus P˙ abc

= =

5 6

1 (b − a) × (c − a) 2

    1 ) A = P · n  A = P · (b − a) × (c − a) (f · n 2 P · (a × b) + P · (b × c) + P · (c × a)

¨ In this book, we avoid nonorientable surfaces such as Mobius strips. A position vector r = rx δx + ry δy + rzδz is a vector that starts at the origin and ends at the point defined by (x, y, z) = (rx , ry , rz). See Section 4.6 for more applications of position vectors.

4.3 Tensor Algebra

Remarks: If we generalize the property flow vector P instead to a vector field P (x, y, z) and call it the flux vector field, then the other usage of the term flux, say,  f , for example, as used in electromagnetics, is the integral of the dot product of the vector field with the unit vectors normal to the surface at the different points, that is,    (x, y, z)dS f = P (x, y, z) · n S

This results in a scalar quantity with units of the property per unit time – no longer per unit area. Hopefully, the context of the application will be sufficient to distinguish the two meanings of flux.

Among the different possible dyads, an important set of nine dyads based on the Cartesian unit vectors is the set of unit dyads, given by     δx δ y , δx δz ( δx δx ),       δy δ x , δy δy , δy δz     δZ δy , δZ δz ( δZ δx ), The concept of dyads can be extended to triads, and so forth, collectively known as polyads. Definition 4.2. An nth -order polyad based on vectors v1 , . . . , vn denoted by (vn · · · v1 ) is a multilinear functional operator7 (based on dots products) acting on a given sequence of n-input vectors x1 , · · · , xn to yield a scalar output, as follows:    n (4.14) (vn · · · v1 ) x1 , · · · , xn = (vi · xi ) i=1

Note that dyads are second-order polyads. With respect to (4.13), (w · x) v can continue to operate on another input vector, say, a, such that   (vw) x, a = (w · x) (v · a) Based on the three Cartesian unit vectors, there will be 3n number of n th -order th polyads, which we refer to as the  unit n -order polyads. For example, there will be 33 unit triads such as ( δx δx δx ), δx δx δy , and so forth. An n th -order tensor is defined as: ) * Definition 4.3. Let B = δ1 , · · · , δm be a set of orthonormal basis vectors of an m dimensional space S. The nth - order tensor under B is a linear combination of n th order unit polyads formed by the unit basis vectors in B. 7

For f to be a multilinear functional operator, we mean that f (x, . . . , y, αu + βv, w, . . . , z) = αf (x, . . . , y, u, w, . . . , z) + βf (x, . . . , y, v, w, . . . , z) where α and β are scalars.

159

160

Vector and Tensor Algebra and Calculus

Figure 4.6. Decomposition of stress τ into a normal stress, τ n , and shear stress, tau s , with respect to plane S at point p .

The zeroth order tensors are scalars, and first-order tensors are vectors.8 By convention, the term “tensors” refers to second-order tensors. Based on this convention, we denote tensors by a letter with double underlines. We also limit our discussion to spaces with a maximum of three dimensions. Using the three Cartesian unit vectors, we have       T = T xx (δx δx ) + T xy δx δy + T xz δx δz + · · · + T zz δzδz (4.15) where the scalar T ij is called the (i, j )−component of tensor T . A special tensor, called the unit tensor, is defined by δ = δx δx + δy δy + δzδz

(4.16)

When this tensor operates on any vector, the result is the same vector.

Stress Tensors Consider a material contained in a 3D domain. Stress is defined as the amount of force F , applied at a point p , per unit area of a fixed plane S that includes the point p . In general, the force F may be on an oblique angle θ with the plane S. We can identify the plane S by a unit vector n (S) that is perpendicular to S, known as its unit normal vector. Then the stress τ (p, S) ( i.e., at p with respect to plane S ) can be decomposed into two additive vectors, EXAMPLE 4.4.

τ (p, S) = τn (p, S) + τs (p, S) where τ n (p, S), called the normal stress, is pointed along the direction of the unit normal vector, and τs (p, S), called the shear stress, is pointed along a direction perpendicular to the unit normal vector (see Figure 4.6). If the material is nonuniform in terms of properties and applied forces, then the stress vector needs to be localized by using infinitesimal planes dS instead of S. The vector τ is also known as the traction vector. 8

A vector v can be considered a tensor only when it acts as an operator on another vector, say, a (which in our case is via dot products),   v a =v·a Otherwise, a vector is mostly just an object having a magnitude and direction. Also, the n th -order tensor described in Definition 4.3 is still limited to the space of physical vectors, and the functional operations are limited to dot products. A generalization to abstract linear vectors is possible, but requires an additional vector space called the dual vector space.

4.3 Tensor Algebra

Figure 4.7. The stress vector with respect to δz as a sum of the normal stress and shear stress.

By representing vectors in the Cartesian coordinate system, we can define a stress tensor as a mapping that contains the state of the stress of a material at a point p . Let T be the stress tensor given by T

=

T xx δx δx + T xy δx δy + T xzδx δz +T yx δy δx + T yy δy δy + T yzδy δz +T zx δzδx + T zy δzδy + T zzδzδz

Let us explore the coefficients of T . Consider the (x, y) plane at point p (see Figure 4.7). The unit normal vector is δz. When tensor T operates on δz, τ(p,δ ) = T δz = T xzδx + T yzδy + T zzδz z

T zz is the term along δz, and it yields the normal stress with respect to the (x, y) plane. Conversely, the other two terms, T xz and T yz, are the x and y components of the shear stress. For the general case, we can represent the unit normal vector n as n = n x δx + n y δy + n zδz The stress (or traction) vector can then be obtained by letting T operate on n,   τ(p,n) = T n (4.17) = =

   T xx δx δx + T xy δx δy + . . . + T yzδy δz + T zzδzδz n x δx + n y δy + n zδz     T xx n x + T xy n y + T xzn z δx + T yx n x + T yy n y + T yzn z δy   + T zx n x + T zy n y + T zzn z δz

Note that to determine the normal stress vector we need to project this vector along the direction of n. (see Example E4.7). Equation (4.17) is a statement of the relationship between the stress tensor and the stress vector and is known as Cauchy’s fundamental theorem for stress.9

9

In some books on transport phenomenon or continuum mechanics, the position of the tensor elements are switched (i.e., the transpose of T described previously) yielding the traction vector as τ = n · T.

161

162

Vector and Tensor Algebra and Calculus

Using the properties of dot products and Definition 4.1, we have the following tensor operations:      Tensor Addition: T +S = T ij + Sij δi δ j i=x,y,z j =x,y,z

Scalar Multiplication αT

of a Tensor:

=





i=x,y,z j =x,y,z

Inner Product of Two Tensors:

T ·S

=





   αT ij δi δ j ⎛ ⎝

i=x,y,z j =x,y,z

Inner Product of a Tensor with a Vector:

T ·v

=

 i=x,y,z

Double Dot Product:

T : S

=



⎛

⎞

  T ik Skj ⎠ δi δ j

k=x,y,z

⎞



⎝



T ij v j ⎠ δi

j =x,y,z



(T ij Sji )

i=x,y,z j =x,y,z

(4.18)

4.4 Matrix Representation of Vectors and Tensors In using the Cartesian coordinates, the tensor and vector operation can also be represented by matrices. First, the unit vectors are represented by: ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ 1 0 0 δx = ⎝ 0 ⎠ δy = ⎝ 1 ⎠ δz = ⎝ 0 ⎠ (4.19) 0 0 1 With (4.19), correspondences can be established between tensor operations and matrix operations. These are given in Table 4.4. Once the vectors and tensors have been translated to matrix equations, the various properties of matrices apply. For instance, the eigenvectors of a symmetric tensor will be orthogonal to each other, associated with three real eigenvalues. Moreover, if the symmetric tensor is positive semi-definite, then eigenvalues are all non-negative real.

EXAMPLE 4.5.

T

=

Consider the stress tensor given by

2δx δx + 2δx δy + δx δz + 2δy δx + δy δy + δy δz + δzδx + δzδy + 4δzδz

It can be represented in matrix form by ⎛ 2 2 T =⎝ 2 1 1 1

⎞ 1 1 ⎠ 4

whose eigenvalues are −0.5688, 2.3644, and 5.2044 and eigenvectors ⎛ ⎛ ⎛ ⎞ ⎞ ⎞ 0.6035 0.6228 0.4479 vλ=−0.5688 = ⎝−0.7962⎠ : vλ=2.3644 = ⎝ 0.4377⎠ ; vλ=5.2044 = ⎝0.4176⎠ 0.0422 −0.6484 0.7601

4.4 Matrix Representation of Vectors and Tensors Table 4.4. Correspondence between tensors and matrices Tensor notations

Matrix representation

αv

T  a = ax , ay , az T  a + b = ax + bx , ay + by , az + bz T  α v = αvx , αvy , αvz

a·b

aT b

ab

abT

a = ax δx + ay δy + azδz a+b

a×b where H[a] T =





 tij

δi δ j



i=x,y,z j =x,y,z

H[a] b    [a] [a] = hij ; hij = 3k=1 i,j,k ak   T = tij   T + S = tij + sij

T +S αT

α T = (αtij ) ⎛ ⎞  T S=⎝ tik skj ⎠

T ·S

k=x,y,z

⎛

T ·v

T v=⎝

 j =x,y,z

T : S

txj v j ,



tyj v j ,

j =x,y,z



⎞T tzj v j ⎠

j =x,y,z

trace(T S)

If desired, these orthonormal eigenvectors can be put back in vector notation, for example, vλ=−0.5688 = 0.6035δx − 0.7962δy + 0.0422δz These eigenvectors determine the principal axes of the tensor, which can be represented by an ellipsoid, as shown in Figure 4.8. Along the lines of the eigenvectors, there are no shear stresses, just normal stress. The eigenvector corresponding to the eigenvalue with the largest magnitude determines where the maximum normal stress is directed and is shown in the figure as line va, whereas the minimum normal stress occurs along the line shown as line vb. Along these axes, if the eigenvalue is positive, we can consider it as tension; otherwise, if the corresponding eigenvalue is negative, we have compression.10 Thus along va we have tension, and along vb we have compression.

Remarks: The items in the right column of Table 4.4 can be translated directly to matrix operation commands in MATLAB. In addition, MATLAB provides some 10

As usual, the sign convention for tension and compression can be switched depending on which field of study defines the tensors.

163

164

Vector and Tensor Algebra and Calculus va

4 z

2 0

vb

-2 -4 2 0 y -2

-2

0

Figure 4.8. Stress tensor ellipsoid. The line along va has the maximum normal stress, whereas the line along vb has the minimum normal stress.

2 x

vector operation commands such as q=dot(A,B) and C=cross(A,B) to evaluate dot products and cross products of matrix A and B, respectively, where each column of A[=]3 × M and B[=]3 × M contains the vectors being operated on.

4.5 Differential Operations for Vector Functions of One Variable Consider a vector fixed at a point, say, p, but is changing its direction and magnitude as a function of variable t. Then we can define a vector derivative as

Definition 4.4. The derivative of v(t) with respect to t, denoted by dv/dt, is given by d 1 v = lim

t→0 t dt





v (t + t) − v (t)

(4.20)

In Cartesian coordinates, the vector v(t) = vx (t)δx + vy (t)δy + vz(t)δz will have a derivative given by d v(t) dt

=

1 lim

t→0 t



  vx (t + t)δx + vy (t + t)δy + vz(t + t)δz   − vx (t)δx + vy (t)δy + vz(t)δz

=

dvy dvx dvz δ + δ + δ dt x dt y dt z



(4.21)

A geometric interpretation of (4.21) is shown in Figure 4.9. Note that the vector dv/dt does not necessarily point in the same direction as v, and neither does it have the same physical units as v (unless t is unitless). Using (4.21) together with the properties of vector operations given in Table 4.2, the following identities can be obtained:

4.6 Application to Position Vectors

165

Figure 4.9. Definition of dV/dt.

Sums: Scalar Products: Dot Products: Cross Products:

 d  u(t) + v(t) dt  d  ψ(t)v(t) dt  d  u(t) · v(t) dt  d  u(t) × v(t) dt

=

du dv + dt dt

=

dψ dv v+ψ dt dt

=

du dv ·v+u· dt dt

=

du dv ×v+u× dt dt

(4.22)

Note that the order is important for the derivatives of cross product because cross products are anti-commutative.

4.6 Application to Position Vectors One of the most useful vectors is the position vector, which we denote by r r = xδx + yδy + zδz If x(t), y(t) and z(t) are functions of a real parameter t, then a path is traced as t increases from a starting value t0 to a final value tF . If the path begins and ends at the same point, that is, x (t0 ) = x (tF ), y (t0 ) = y (tF ), and z (t0 ) = z (tF ), then we say that the path is a closed path. If the path does not intersect itself before the final point, then we call it a simple path. The parameter t is often chosen to be time or elapse time. If we now apply the derivative operation to the position vector, we have the velocity vector, v (t),11 v(t) = vx δx + vy δy + vzδz =

d dx dy dz r = δ + δ + δ dt dt x dt y dt z

where the components are vx = dx/dt, vy = dy/dt and vz = dz/dt. The norm of v(t) is the speed, denoted by s˙ , " s˙ (t) = v(t) = v2x + v2y + v2z 11

(4.23)

(4.24)

We reserve the arrow sign above the variable to signify the derivatives of position vectors, including velocity, acceleration, and so forth, and their components.

166

Vector and Tensor Algebra and Calculus

In as much as the position (x, y, z) changes with the parameter t, the distance traveled along the path will also increase. We call this distance the arc length at t, denoted by s(t) and defined as  t v (τ) dτ (4.25) s(t) = t0

Note that the integrand is monotonic and nondecreasing, which means s(t) ≥ 0. Differentiation will yield back the speed, that is, " ds = v(t) = s˙ (t) = v2x + v2y + v2z dt Vector v(t) will be oriented tangent to the path at t. Thus to find the unit tangent vector at s, denoted by t(t), a normalization of v(s) is sufficient, v(t)

t(t) =

(4.26)

v(t)

A derivative of t(t) with respect to t is also possible. Because t(t + dt) and t(t) both have magnitude equal to 1, the only change will be in the direction. Using this fact, we can apply the formula for the derivative of dot products given in (4.22), t(t)

2

= t · t = 1

→

d   t · t dt   dt 2 t · dt

=

0

=

0

Thus dt/dt is perpendicular to t. If we normalize this new vector to have a magnitude  , that is, of 1, we obtain the unit normal to the curve at t, denoted by n = n

dt/dt dt/dt

(4.27)

 (t), we can find a third unit vector, called the binormal unit Once we have t(t) and n  , that is perpendicular to both t and n  defined by vector, denoted by b  = t × n  b

(4.28)

 /dt should yield information about how fast the shape of The vectors dt/dt and db the path is changing such as curvature and torsion, but they are incomplete because both vectors are rates with respect to t, which is not ideal if only the shape is needed. For the purpose of measuring the curvature of the path, one choice is instead to take the ratio of the change in t(t) per change in arclength s(t), dt dt/dt = ds ds/dt The magnitude of this ratio is a measure known as the path curvature at t and denoted by κ(t), κ(t) =

dt/dt ds/dt

=

dt/dt s˙ (t)

4.6 Application to Position Vectors

167

The radius of curvature, denoted by rcurve , is defined as the reciprocal of the curvature,

rcurve (t) =

1 s˙ (t) = κ(t) dt/dt

This is the radius of a circle that is tangent to the curve at t and two differentially close neighboring points. For the torsion of the path, we have a similar ratio. This time we define torsion of the path at t, denoted by τ(t), as the norm of the change in the binormal unit  (t) per change in arclength s(t), that is, vector b

τ(t) =

 /dt db  /dt(t) db = ds/dt s˙ (t)

The radius of torsion, denoted by rtorsion , is the reciprocal of torsion,

rtorsion =

1 s˙ = τ  /dt db

Finally, we can take the derivative of velocity v(t) to obtain the acceleration vector, denoted by a,

a(t) = ax δx + ay δy + azδz =

dv d2 x d2 y d2 z = 2 δx + 2 δy + 2 δz dt dt dt dt

(4.29)

where the components are given by ax = d2 x/dt2 , ay = d2 y/dt2 and az = d2 z/dt2 .  as follows, Alternatively, we could represent acceleration in terms of t and n

a(t)

 aT t + aN n

=

  d s˙t dv = dt dt

=

d˙s dt d˙s dt t + s˙ = t + s˙  n dt dt dt dt

=

d˙s t + s˙ 2 κ n dt

Thus the tangential and normal components of the acceleration vector are given by aT = d˙s/dt and aN = s˙ 2 κ, respectively.

168

Vector and Tensor Algebra and Calculus

Consider the helical path described by x(t) = cos(t), y(t) = sin(t) and z(t) = t. Then we have the following:

EXAMPLE 4.6.

Position Vector:

r

=

cos(t)δx + sin(t)δy + tδz

Velocity:

v

=

Speed:

s˙

=

− sin(t)δx + cos(t)δy + δz √ 2

Unit Tangent:

t

=

√ δ + − sin(t) 2 x

Unit Normal:

 n

=

− cos(t)δx − sin(t)δy

Unit Binormal:

 b

=

sin(t) √ δ 2 x

Curvature:

κ

=

1/2

Torsion:

τ

=

1/2

Acceleration:

a

=

 − cos(t)δx − sin(t)δy or n

−

cos(t) √ δ 2 y

cos(t) √ δ 2 y

+

+

√1 δ 2 z

√1 δ 2 z

Let a 3D surface S(x, y, z) = 0 be parameterized by ξ and η, x = x (ξ, η) ; y = y (ξ, η) ; z = z (ξ, η) ) * Then to find the normal to the surface S at a point a = ax , ay , az , we could use the previous results to first find a vector that is tangent to the curve in S at point a along constant ξ and another vector that is tangent to the curve in S at point a along constant η. Thus tξ

=

∂r ∂x ∂y ∂z = δ + δ + δ ∂ξ ∂ξ x ∂ξ y ∂ξ z

tη

=

∂r ∂x ∂y ∂z = δx + δy + δz ∂η ∂η ∂η ∂η

Then we take the cross product of both tangent vectors to find a vector that is perpendicular to the plane that contains both tangent vectors and then normalize the result to find a unit normal vector, = n

tξ × tη tξ × tη

(4.30)

(See Exercise E4.10 as a sample problem.) Another approach to finding the unit normal vector to the surface is to use the gradient function. This is discussed in the next section.

4.7 Differential Operations for Vector Fields

4.7 Differential Operations for Vector Fields Let ψ (p) be scalar field that is a function of the spatial position p. We assume that the scalar fields will be sufficiently differentiable. Likewise, let v (p) be a vector field that is a function of position p. We assume that the components of v are sufficiently differentiable. Under the Cartesian coordinate system, this means v (x, y, z) = vx (x, y, z) δx + vy (x, y, z) δy + vz (x, y, z) δz where vx (x, y, z), vy (x, y, z), and vz (x, y, z) are scalar fields that are assumed to be sufficiently differentiable. In the following sections, we first discuss the various differential operations of vector fields based on the Cartesian coordinate systems. Later, we show how these differential operations can be defined in other coordinate systems, such as cylindrical and spherical coordinates systems. Remarks: In MATLAB, the commands quiver and quiver3 plot arrows depicting the vector field at the specified mesh positions.

Let v(x, y, z) be a given velocity field of a fluid domain. A set of curves known as streamlines can be found such that at any point, say, p = (x∗ , y∗ , z∗ ), in a streamline, v(p) will be tangent to that curve at p. If the fluid flow is changing with time, the snapshots of the collection of streamlines will also change. However, under steady state, the streamlines will show the path lines of fluid particles. Thus streamlines offer another view of fluid motion; that is, instead of a distribution of arrows, we have a portrait containing several curves. Let the position of a streamline curve be parameterized by s, that is, r = r(s); then we could use the parallelism property of cross products to obtain the condition dr × v(r(s)) = 0 (4.31) ds This condition is satisfied by the following equalities: EXAMPLE 4.7.

dx dy dz = vx ; = vy ; = vz (4.32) ds ds ds subject to initial conditions r(0) = r0 . However, in general, these simultaneous differential equations may not be that easy to evaluate. One could use numerical ODE solvers, discussed in Chapter 7. In MATLAB, the command streamline uses finite difference approximations to obtain the streamlines by specifying the starting points. Another MATLAB command streamtube adds tube-width information to show the magnitude of the velocity instead of a simple curve. To illustrate, consider a 2D velocity field described by     17 4 29 5 2 x −y+ δx + − x2 + (y − x) − δ v= 3 6 3 30 y We can plot the velocity field and streamlines as shown in Figure 4.10. This figure was generated using a MATLAB file that are available in the book’s webpage, streamline_test([0 0 0.04 0.1 0.18 0.3], ([2.05 2.2 2 2 2 2],0,1,2,3)

169

170

Vector and Tensor Algebra and Calculus

3

y

2.5

Figure 4.10. A velocity field together with some streamlines.

2

0

0.5 x

1

4.7.1 Gradients Consider a 3D scalar field ψ(x, y, z). For example, let u(x, y, z) be a scalar temperature field, that is, a temperature distribution in the 3D space. Usually, the gradation of scalar fields will determine the driving forces in mass, momentum, or energy transport. Then it is important to determine a vector field based on the gradations of the scalar field that can be used to determine the directions (and magnitude) of driving forces. The desired vector field is known as the gradient vector field. The gradient of ψ(x, y, z) is then defined as gradient(ψ(x, y, z)) =

∂ψ ∂ψ ∂ψ δx + δy + δ ∂x ∂y ∂z z

(4.33)

This assumes that the scalar field is continuous and smooth (i.e., differentiable in each independent variable). The gradient definition can be simplified by introducing a differential operator called the gradient operator, or grad for short.12 The grad operator is denoted by ∇, and it is defined as ∇ = δx

∂ ∂ ∂ + δy + δz ∂x ∂y ∂z

(4.34)

Thus (4.33) can be rewritten as gradient(ψ(x, y, z)) = ∇ψ(x, y, z) EXAMPLE 4.8.

Consider the following scalar field, which is independent of z,

given by 2 2 ψ(x, y) = e−(x +y )

then the gradient is given by     2 2 2 2 ∇ψ (x, y) = −2xe−(x +y ) δx − 2ye−(x +y ) δy A plot of ψ(x, y) is given in Figure 4.11. Let us look at the gradient at position (x, y) = (−0.5, −1). The x-component of the gradient is given by the slope of a curve at y = −1, as shown in the figure. The result is ∂ψ/∂x = 0.287. The y-component of the gradient is given by the slope of a curve at x = −0.5, also shown in the figure. The resulting value of the y-component of the gradient is 12

In some texts, it is also called the “del” operator.

4.7 Differential Operations for Vector Fields

171

1 ψ(x,y)

0.5

0 2 1 0 -1 y

-2

-2

0

-1

1

2

x

Figure 4.11. The surface plot of ψ(x, y), together one curve at constant y = −1 and another curve at constant x = −0.5.

∂ψ/∂y = 0.573 (about twice as steep as the other slope). The gradient is given by ∇ψ (−0.5, −1) = 0.287δx + 0.573δy whose magnitude is ∇ψ (−0.5, −1) = 0.641. If we had chosen the point (x, y) = (−1.5, −1), the resulting gradient would be ∇ψ (0, −1) = 0.116δx + 0.078δy where the x-component is about twice as large as the y-component, and the magnitude of the gradient vector is ∇ψ (−1.5, −1) = 0.14. One could look at Figure 4.11 and see that indeed the gradient at the point (x, y) = (−0.5, −1) is much steeper than (x, y) = (−1.5, −1). Furthermore, one can see that the gradient is pointing at an ascending direction.

The gradient vector could also be used to obtain a directional derivative of a 3D scalar field. Definition 4.5. For a multivariable function f (x1 , . . . , xn ) and normalized vector v = (v1 , . . . , vn ), the directional derivative of f along unit vector v, denoted by Dv f (x), is defined as Dv f (x) = lim+ α→0

f (x + αv) − f (x) α

(4.35)

For ψ (x, y, z), the numerator of (4.35) sets x = αvx , y = αvy , and z = αvz. Then for small changes,

ψ =

∂ψ ∂ψ ∂ψ ∂ψ ∂ψ ∂ψ

x +

y +

z = αvx + αvy + αvz ∂x ∂y ∂z ∂x ∂y ∂z

172

Vector and Tensor Algebra and Calculus

After substitution to (4.35) and taking the limit, we have Dv ψ (x, y, z) =

∂ψ ∂ψ ∂ψ vx + vy + vz = ∇ψ · v ∂x ∂y ∂z

(4.36)

The directional derivative is the scalar projection of the gradient along the direction of vector v. Based on the definition (4.35), the value of the directional derivative is zero when ψ is a constant along v. For a 2D case, for example, ψ = ψ(x, y) = κ, this defines a curve (possibly closed) called the contour curves or contours of ψ (x, y) at different levels determined by the parameter κ. However, for a 3D scalar field, that is, ψ = ψ (x, y, z) = κ, these surfaces are called the contour surfaces or isosurfaces determined by the parameter κ. Because the directional derivative is zero along the contour lines or contour surfaces, we see from (4.36), that the gradient ∇ψ is perpendicular to either the contour lines for the 2D case or the contour surfaces for the 3D case. Remarks: In MATLAB, the commands contour and contour3 gives the contour plots of a surface, for example, of a 2D scalar field. For a 3D scalar field, one can instead use contourf to obtain contour slice plots or use the command isosurface to find the contour surfaces at fixed values (one can then use the command hold to allow several surface plots while introducing NaN’s to produce cutouts of the surfaces). For the gradients, there is the MATLAB command gradient, which calculates the gradient based on finite differences; but the accuracy depends on the resolution of the mesh. The result of the gradient can then be plotted using quiver or quiver3. Suppose the dimensionless temperature distribution u = (T − T center )/T center is given by the following scalar field:  x 2 u= + y2 + z2 3 The contour surfaces, that is, u = κ=constant, are ellipsoids centered at the origin. The surfaces corresponding to κ = 0.5, 1.0, 1.5 are shown in Figure 4.12. Let use consider two points at the surface defined by κ = 1: a = (2.9, 0, 0.2560) and b = (0, 0, 1). At point a, we find that the gradient is given by ∇u|a = 0.644δx + 0.5121δz and ∇u|a  = 0.823. At point b, we have the gradient ∇u|b = 2δz and ∇u|b = 2. Both gradients can be seen to be normal to the contour surfaces. Note that the distance between the surface at κ = 1 and κ = 1.5 around point a is farther than the distance between surfaces around point b, and yet the magnitude of the gradient at point a is less than half the magnitude of the gradient at point b. Thus one should be cautious when reading contour maps or contour surfaces; the closer the adjacent contours are to each other, the greater the magnitude of the gradient. EXAMPLE 4.9.

The gradient vector can also be used to determine the rate of change in ψ along a path C. The differential of ψ(x, y, z) is given by dψ(x, y, z) =

∂ψ ∂ψ ∂ψ dx + dy + dz ∂x ∂y ∂z

4.7 Differential Operations for Vector Fields

173

2 1

Figure 4.12. The contour surfaces of u at κ = 0.5 (innermost ellipsoid), κ = 1.0 (middle ellipsoid), and κ = 1.5 (outermost ellipsoid). Also, the gradients at points a and b are shown to be perpendicular to the surface corresponding to κ = 1.

b

z0 -1

a

-2 -5 0 5

-2

x

y

Along the path, the rate of change in ψ per change in arclength s is given by dψ(x, y, z) ∂ψ dx ∂ψ dy ∂ψ dz = + + ds ∂x ds ∂y ds ∂z ds

(4.37)

The right-hand side of the equation can now be factored into a dot product,   dψ(x, y, z) dx dy dz = (∇ψ) · δ + δ + δ ds ds x ds y ds z =

(∇ψ) · t

(4.38)

where t is the unit tangent vector ( cf. (4.26)). Thus the rate of change of ψ along the path C is the directional derivative along the unit tangent vectors of the path at the desired point. Let θ be the angle between ∇ψ and t. Then (4.38) becomes dψ = ∇ψ cos θ ds

(4.39)

This means that at a point, the maximum value for increase of ψ occurs when cos θ = 1 or θ = 0, whereas the maximum rate of decrease of ψ occurs when cos θ = −1 or θ = π. This generates one of the methods used for the search for local optima of ψ (x, y, z), that is, to choose an update path C directed along the gradient ∇ψ to find the local maximum, called the gradient ascent method. In other cases, we choose an update path negative to the direction of ∇ψ to find a local mimimum, which is called the gradient descent method. If the rate of change, dψ/ds, at a point is zero for all θ, that is, for all paths passing through this point, then the gradient must be zero at this point. This indicates an extreme point or critical point, which is either an optimum or a saddle point. Thus a necessary condition for optimality is ∇φ = 0. Finally, the gradient operation can be used to find the unit normal of any smooth surface. Let the given surface be given as f (x, y, z) = C. As an alternative to methods used in Section 4.6, for example, (4.30), the unit normal to f (x, y, z) = C at the point a = (ax , ay , az) is given by n=

∇f ∇f

2

0

(4.40)

174

Vector and Tensor Algebra and Calculus

Figure 4.13. Evaluation of the divergence of a vector field at a point.

4.7.2 Divergence Consider the vector field v (x, y, z), v(x, y, z) = vx (x, y, z)δx + vy (x, y, z)δy + vz(x, y, z)δz The divergence of a vector field at a point is defined as   ∂v ∂vy ∂vz x divergence v (x, y, z) = + + ∂x ∂y ∂z

(4.41)

In terms of the ∇ operator, the divergence is written as ∇ · v, ⎛ ⎞ ⎛ ⎞    ∂v j ∂ ∇ ·v = ⎝ δi ⎠ · ⎝ vj δj ⎠ = δ ·δ ∂i ∂i i j j =x,y,z i=x,y,z i,j =x,y,z =

 ∂vi ∂vy ∂vx ∂vz = + + ∂i ∂x ∂y ∂z i=x,y,z

Based on this definition, the evaluation of divergence is shown in Figure 4.13. The divergence of a vector field yields scalar fields. This is in contrast to the gradient of a scalar field, which yields a vector field. For instance, suppose the vector field is a flux field, that is, the rate of transport of a quantity (momentum, mass or energy) per unit area across the flow. The divergence of the flux field at a point will then measure how much the transported quantity will “diverge” at that point due to the field. Specifically, for mass flux, we can write the continuity equation (or differential mass balance) as   ∇ · ρ(x, y, z)v(x, y, z) = 0 where ρ is the density and v is the velocity field. Note that in the most general case, both ρ and v will depend on the position. A vector field v is called solenoidal if ∇ · v = 0.

4.7 Differential Operations for Vector Fields

175

2

1.5

2 1

0.5

1

∇⋅v

z 0

0

−0.5

−1

−1 2

−1.5

2 0

y

−2 −1

0 −2 −2

−0.5

0

0.5

1

z

x

Figure 4.14. The plot of the vector field v with f (z) = z and the divergence ∇ · v = 2z.

Remarks: In MATLAB, the command divergence calculates the divergence field based on finite difference. The result can then be plotted using surf or surfl for 2D fields. For 3D fields, one can use the command slice to obtain a set of color/shade graded slices. EXAMPLE 4.10. Consider the vector field, v = (xf (z)) δx + (yf (z)) δy + δz. The divergence is then given by ∇ · v = 2f (z). Thus, for this example, the divergence yields a scalar field that is dependent only in z. Let us now look at two cases. First, consider f (z) = z. The vector field is shown in Figure 4.14. The divergence varies linearly with z, and one can see that the divergence is positive when z > 0 and the field appears to be diverging, whereas the divergence is negative when z < 0 and the vector field appears to be converging. √ For the second case, consider f (z) = z2 . The vector field is shown in Figure 4.15. As the vector field shows, there appears to be no region where the vector field is converging; thus the divergence is always positive, except at z = 0.

2

2 1.5

1

∇⋅v

z 0

−1 2

1

0.5

2 0

y

0 −2 −2

x

0 −1

Figure 4.15. The plot of the vector field v with f (z) =

√

−0.5

0

z

0.5

√ z2 and the divergence ∇ · v = 2 z2 .

1

176

Vector and Tensor Algebra and Calculus

Figure 4.16. Evaluation of the curl of a vector field at a point across the (x, y) plane.

4.7.3 Curl Due to the distribution in a vector field, neighboring vectors will affect the differential volume at a point to twirl  In Figure 4.16, for a particle (i.e., a differential  or rotate. volume) at point P = Px , Py , Pz , we consider the projection of the vector fields in the (x, y) plane at Pz level. Let Czδz be a vector that describes the tendency of the particle at P to rotate based on the relative vector distributions of vx (x, y, Pz) along y-direction and the relative vector distribution of vy (x, y, Pz) along the x-direction. Using the right-hand screw convention, the total effect is given by   vy (x + x, y, z) − vy (x, y, z) vx (x, y + y, z) − vx (x, y, z) Cz = lim −

→0

x

y =

∂vy ∂vx − ∂x ∂y

Following the same analysis in the (x, z) plane, we get Cy δy , where Cy =

∂vx ∂vz − ∂z ∂x

and in the (y, z) plane, we have Cx δx , where Cx =

∂vz ∂vy − ∂y ∂z

Adding the three vectors yields a vector known as the Curl of the vector field at the point (x, y, z),       ∂vy ∂vz ∂vy ∂vz ∂vx ∂vx − δx + Curl(v(x, y, z)) = − δy + − δz (4.42) ∂y ∂z ∂z ∂x ∂x ∂y

4.7 Differential Operations for Vector Fields

177

Using the ∇ operator, the curl is written as the cross product of ∇ with v, ⎛ ⎞ ⎛ ⎞    ∂v j  ∂v j ∂ ∇ ×v = ⎝ δi ⎠ × ⎝ vj δj ⎠ = δi × δ j = ijk δ ∂i ∂i ∂i k j =x,y,z i=x,y,z ij 

=

∂vz ∂vy − ∂y ∂z





δx +



∂vx ∂vz − δy + ∂z ∂x



ijk

∂vy ∂vx − ∂x ∂y



δz

(4.43)

With the rectangular coordinates, each unit vector will have the same direction and magnitude at any point in space. Under the Cartesian coordinate system, the following mnemonic is valid: ⎛ ⎞ δx δy δz ⎜ ⎟ ⎜ ∂ ∂ ∂ ⎟ ⎜ ⎟ ∇ × v = det ⎜ ⎟ ∂x ∂y ∂z ⎝ ⎠ vx

vy

vz

For the special case in which v is a velocity field, ∇ × v is the vector field known as the vorticity of v. The angular velocity of a rigid body is related to the curl by

=

1 ∇ ×v 2

(4.44)

A vector field v is called irrotational if ∇ × v = 0. Remarks: In MATLAB, the command curl can be used to generate the curl velocity field. Another MATLAB command streamribbon can be used to attach the curl information by the twisting of a ribbon attached to the streamline in which the width of the ribbon represents the magnitude of the velocity at that point.

Consider the following vector field:     y x ! ! v= − δx + δy + δz 0.1 + x2 + y2 0.1 + x2 + y2

EXAMPLE 4.11.

then the curl is given by

⎛

⎞ ! 1 + 5 x2 + y2 ⎟ ⎜ ∇ × v = ⎝20   2 ⎠ δz ! 1 + 10 x2 + y2

The vector field is shown in Figure 4.17, where the flow appears to follow a helical path. The curl is also a vector field, but for this particular example, it turns out to only have a z-component that is independent of z. Thus we can plot the curl as as function only of x and y, which is also shown in Figure 4.17. From the figure, one sees that the curl increases radially outward in any (x, y) plane. Also, note that none of the vectors located far from the z-axis are lying in a plane that is parallel to the (x, y) plane, and yet the curl at all points are directed in the positive z-direction. This shows that the curl is not necessarily perpendicular to the vector v at that point. This is because the curl is a differential operation on a vector field and not a cross product of two vectors.

178

Vector and Tensor Algebra and Calculus

2 1

1

⎢⎢∇ × ν ⎢⎢

z 0

0.5

−1 2

0 1 1

y

0

0

−2 −2

0

−1

1

0

2

y

–1 –1

x

x

Figure 4.17. The plot of the vector field v and ∇ × v .

As shown in Example 4.11, the curl is not necessarily perpendicular to the vector field. However, there are situations in which the curl is perpendicular to the vector field. If v · (∇ × v) = 0, then v is known as a complex lamellar vector field. An example of a complex lamellar vector field is v = vx (x, y) δx + vy (x, y) δy + 0δz Conversely, there are also vector fields whose curls are always parallel to the vector field. If v × (∇ × v) = 0, then v is known as a Beltrami vector field. An example of a Beltrami vector field is     v = α sin z + β cos z δx + −β sin z + α cos z δy + 0δz where α and β are constant.

4.7.4 Laplacian As we discussed earlier, the gradient of a scalar field ψ(x, y, z) will result in a vector field ∇ψ. By taking the divergence of ∇ψ, we obtain the Laplacian of ψ(x, y, z), Laplacian(ψ(x, y, z))

=

∇ · (∇ψ)

(4.45)

=

∂2ψ ∂2ψ ∂2ψ + + ∂x2 ∂y2 ∂z2

(4.46)

The Laplacian operator is often given the shorthand notation of ∇ 2 to mean ∇2 = ∇ · ∇

(4.47)

Thus the Laplacian of ψ is commonly denoted as ∇ 2 ψ. As a simple example, consider the heat conduction process of a 3D solid; we can take the energy balance to be given by Rate of Change in Energy in Control Volume = Negative Divergence of Energy Flow

4.7 Differential Operations for Vector Fields

The energy flow following Fourier’s law is proportional to the negative gradient of temperature, that is, energy flow = −k∇T , or (divergence of energy flow) = ∇ · (−k∇T ). The rate of energy is given by d (ρCp T ) /dt. With constant heat conductivity, density, and heat capacity, the energy balance reduces to ∂T = α∇ 2 T ∂t where α = k/(ρCp ), k is thermal conductivity, ρ is density, and Cp is the heat capacity. (See Section 5.4.1 for a more detailed derivation that involves the divergence theorem.)

EXAMPLE 4.12.

Consider the following 2D temperature scalar field 2 2 T (x, y) = e−(x +y )

then

    ∇T = −2T xδx + yδy and ∇ 2 T = 4 r2 − 1 T ! where r = x2 + y2 . Thus the Laplacian of T is a negative factor of T when r < 1, zero when r = 1, and a positive factor of T when r > 1. The scalar field, gradient field, and Laplacian field are shown in Figure 4.18. One can see that the gradient field is directed toward the center and zero at the origin. Because the gradients are all pointing toward the origin around the origin, a negative divergence is expected there, and this can be seen for the the Laplacian. Scalar functions ψ that satisfy ∇ 2 ψ = 0 in a region S are known as harmonic functions. Harmonic functions in a closed region S have special direct relationships with the values of ψ at the boundaries of S, and these relationships can be derived using the divergence theorem and Green’s identities, which are topics in the next chapter. The Laplacian operator can also be applied to vectors and tensors, for instance ∇ 2 v = ∇ · ∇v. However, we need to first define ∇v, which is the gradient of a vector. This is discussed in the next section.

4.7.5 Other Differential Operators In addition to the four major differential operators just discussed, we list some additional differential operators: 1. Gradient-Vector Dyad When the gradient operator is combined with a vector v, they form a tensor operator, ⎞   ⎛   ∂vm ∂  ∇v = δk vm δm = ⎝ δ δ ⎠ (4.48) ∂k m ∂k k m k=x,y,z

k,m=x,y,z

with scalar weights ∂vm /∂k for the (δk δm )-component.

179

180

Vector and Tensor Algebra and Calculus 2

1.5

1

1

T 0.5

0.5

y

0

−0.5

0 2

−1

2 0

−1.5

0

y

−2 −2

x

−2

−2

−1

0

1

2

x

2 2

∇ T 0

−2

−4 2

2 0

0 −2 −2

y

x

Figure 4.18. The plot of the temperature field, T (x, y), gradient vector field ∇T , and Laplacian field ∇ 2 T .

If one were to write out the tensor field ∇v as a matrix, we have ⎛

∂vx ⎜ ∂x ⎜ ⎜ ⎜ ∂vx ⎜ [∇v] = ⎜ ⎜ ∂y ⎜ ⎜ ⎜ ∂vx ⎝ ∂z

∂vy ∂x ∂vy ∂y ∂vy ∂z

⎞ ∂vz ∂x ⎟ ⎟ ⎟ ∂vz ⎟ ⎟ ∂y ⎟ ⎟ ⎟ ⎟ ∂vz ⎟ ⎠ ∂z

which is a transposed Jacobian matrix. The inner product of another vector w on ∇v will yield w · ∇v =

 =x,y,z

⎛

w δ · ⎝



k,m=x,y,z

⎞ ∂vm δ δ ⎠= ∂k k m

 m,k=x,y,z

  ∂vm wk δm ∂k

(4.49)

4.7 Differential Operations for Vector Fields

181

2. Dot-Grad We define the dot-grad operator based on w as w · ∇ = wx

∂ ∂ ∂ + wy + wz ∂x ∂y ∂z

(4.50)

Then for scalar field ψ and vector field v, (w · ∇) ψ = wx

∂ψ ∂ψ ∂ψ + wy + wz ∂x ∂y ∂z

and (w · ∇) v = wx

∂v ∂v ∂v + wy + wz = ∂x ∂y ∂z

 k,m=x,y,z

  ∂vm wk δm ∂k

where the last equation is the same as that in (4.49). 3. Cross-Grad The cross-grad operator based on w is defined as       ∂ ∂ ∂ ∂ ∂ ∂ w × ∇ = δx wy − wz + δy wz − wx + δz w x − wy ∂z ∂y ∂x ∂z ∂y ∂x (4.51) for which the mnemonic, applicable only for the rectangular coordinate system, is given by ⎛ ⎞ δx δy δz ⎜ w wy wz ⎟ x ⎟ w × ∇ = det ⎜ ⎝ ∂ ∂ ∂ ⎠ ∂x ∂y ∂z Using the cross-grad operator on a scalar field ψ will yield the following identity, (w × ∇) ψ = w × (∇ψ) Another important identity is obtained when we extend the divergence operation, based on grad operators, to the cross-grad operators, (w × ∇) · v = w · (∇ × v)

(4.52)

4. Laplacian of vector fields The Laplacian was defined earlier as the divergence of a gradient. We can apply the same operation to vectors. Thus, taking the divergence of ∇v, ⎛ ⎞   ∂ ⎝ ∂vm ∇ 2 v = ∇ · (∇v) = δk · δ δ ⎠ ∂k ∂  m ,m=x,y,z k=x,y,z

=

 k,m=x,y,z

=

∂ vm δ ∂k2 m 2

   ∇ 2 vm δm

m=x,y,z

(4.53)

182

Vector and Tensor Algebra and Calculus

Note that the formula given in (4.53) is only for the rectangular coordinates. For other coordinate systems, one could start from the definition of the Laplacian as a divergence of a gradient.

4.7.6 Vector Differential Identities A list of some important identities involving the gradient, divergence, curl, and Laplacian is given next. These identities can be proved by applying the definitions directly to both sides of the equations.   ∇ ψω = ψ∇ω + ω∇ψ (4.54) ∇ · ψv

=

ψ∇ · v + v · ∇ψ

(4.55)

∇ × ψv

=

ψ∇ × v + (∇ψ) × v

(4.56)

∇ · (u × v)

=

v·∇ ×u − u·∇ ×v

(4.57)

∇ × (u × v)

=

v · ∇u − u · ∇v + u∇ · v − v∇ · u

∇(u · v)

=

(4.58)

u · ∇v + v · ∇u + u × (∇ × v) + v × (∇ × u)

(4.59)

∇ × ∇ψ

=

0

(4.60)

∇ ·∇ ×v

=

0

(4.61)

∇ × (∇ × v)

=

(4.62)

v · ∇v

=

∇(∇ · v) − ∇ 2 v    1  ∇ v·v −v× ∇ ×v 2

(4.63)

The first three identities involve operations on scalar products, including product of two scalar fields and the scalar product of a vector field. They are direct results of implementing the properties of derivatives of products. Note that in (4.56), the order of ∇ψ × v is crucial.

Using the divergence theorem, one can derive the conservation of mass (see (5.24)) to be

EXAMPLE 4.13.

∂ρ + ∇ · (ρv) = 0 ∂t where ρ and v are the density field and velocity field, respectively. This can be rewritten in a more familiar form by using the identity (4.55) to replace the second term as follows: ∇ · (ρv) = ρ∇ · v + v · ∇ρ

4.7 Differential Operations for Vector Fields

Then, the conservation equation becomes ∂ρ + v · ∇ρ + ρ∇ · v = 0 (4.64) ∂t or by defining another operator D/Dt(•) known as the substantial rate of change operator (or substantial time derivative) defined by   D ∂ = + v · ∇ (4.65) (•) (•) Dt ∂t the continuity equation (4.64) becomes Dρ + ρ∇ · v = 0 (4.66) Dt For the special case of incompressible fluid (i.e., ρ is constant), (4.66) reduces to ∇ · v = 0.

Equation (4.57) shows that with the gradient operator, the usual cyclicpermutation properties of triple product no longer apply, except when either u or v is constant. Similarly, for (4.58), the identities for the usual triple vector products no longer apply as well. The identity in (4.59) is surprisingly complicated. On the left-hand side is the gradient of a dot product. However, the right-hand side identity includes cross products of curls plus dot products of vectors with gradient-vector dyads. Note that (4.63) is a consequence of (4.59) with v = u. Equation (4.60) states that gradient vector fields are irrotational. However, (4.61) states that curls have zero divergence, which means that curl fields (e.g., vorticity fields) are solenoidal. Both these identities are very useful in solving vector differential equations. For instance, in Navier-Stokes equations, a pressure gradient appears in the momentum balances. By simply taking the curl of the equation, the dependence on pressure disappears because of identity (4.60). Likewise, if one needs to remove curls in an equation, one simply needs to take the divergence of that equation. The identity given by (4.62) relates the curl of a curl with terms involving the Laplacian of a vector field, as well as the gradient of a divergence. In some cases, this formula is used to find the Laplacian of a vector field represented in other coordinate systems. Equation (4.63) can be used to yield an alternative definition for a Beltrami vector field; that is, with v × (∇ × v) = 0, a vector field is a Beltrami field if and only if v · ∇v =

1 ∇ (v · v) 2

Finally, equation (4.63) is also very useful because the term v · ∇v appears in momentum balance equations and is known as the inertial terms. Thus (4.63) is often used to introduce the role of vorticity, ∇ × v, in the equations of fluid dynamics. The proofs of some of the identities can be lengthy but more or less straightforward. The following example shows the proof of the identity given in (4.59).

183

184

Vector and Tensor Algebra and Calculus

Let us prove the identity given in (4.59). We begin with an expansion of the left-hand side of the identity, ⎞ ⎛     ∂ ⎝  ∂uk ∂vk ⎠ ∇ (u · v) = δi uk vk = δi vk + uk (4.67) ∂i ∂i ∂i i=x,y,z

EXAMPLE 4.14.

k=x,y,z

i,k=x,y,z

Next, we show each term in the right-hand side of the identity, ⎛ ⎞   ∂v k u × (∇ × v) = δj uj × ⎝ km δm ⎠ ∂ j =x,y,z m,k,=x,y,z



=

δi ijm km u j

i,j,m,k,=x,y,z

⎛

(u · ∇) v = ⎝



k=x,y,z

   ∂vk ∂vk ∂vi = δi uk − uk ∂ ∂i ∂k i,k

⎞⎛

(4.68)

⎞

   ∂ ⎠⎝  ∂vi ⎠ uk vi δi = δi uk ∂k ∂k i=x,y,z

Combining (4.68) and (4.69), we get u × (∇ × v) + (u · ∇) v =

 i,k

Reversing the role of u and v, we have v × (∇ × u) + (v · ∇) u =

(4.69)

i,k=x,y,z

 i,k

  ∂vk δi uk ∂i

(4.70)

  ∂uk δi vk ∂i

(4.71)

Finally, adding (4.70) and (4.71), we arrive at a sum that is equal to (4.67).

4.8 Curvilinear Coordinate System: Cylindrical and Spherical In this section, we discuss the cylindrical and spherical coordinate systems. The properties and differential operators are summarized in tabular form. The geometric derivations, with the aid of matrix methods, can be found in Sections D.2 and D.3 in the appendix.

4.8.1 Cylindrical Coordinate System The cylindrical coordinate system is a coordinate system shown in Figure 4.19 and defined by the following invertible transformations: ! x = r cos θ r = x2 + y2 −1 ↔ y = r sin θ (4.72) θ = tan (y/x) z = z z = z A summary of the relationships between cylindrical and rectangular coordinates is given in Table 4.5. The derivation of the items in this table can be done via geometric arguments. Details of these derivations, aided by matrix methods, are given in Section D.2.

4.8 Curvilinear Coordinate System: Cylindrical and Spherical

185

Figure 4.19. Cylindrical coordinate system.

The gradients, curls, and Laplacian in cylindrical coordinates have to be evaluated using the definition of the various operators and the distributive properties of dot products and cross products. To illustrate, the divergence formula in cylindrical coordinates can be obtained as follows:     ∂ 1 ∂ ∂ δr + δθ + δz ∇ ·v = · vr δr + vθ δθ + vzδz ∂r r ∂θ ∂z      ∂vr ∂vz 1 ∂  = + δ + · vr δr + vθ δθ ∂r ∂z r θ ∂θ     ∂vr ∂vz 1 ∂vr ∂vθ = + + δθ · vr δθ + δr − vθ δr + δθ ∂r ∂z r ∂θ ∂θ =

∂vr 1 ∂vθ vr ∂vz + + + ∂r r ∂θ r ∂z

(4.73)

Likewise, for the curl,     ∂ 1 ∂ ∂ ∇ ×v = δr + δθ + δz × vr δr + vθ δθ + vzδz ∂r r ∂θ ∂z     ∂vθ ∂vz ∂vr ∂vθ = δz − δθ + δθ − δr ∂r ∂r ∂z ∂z   1 ∂vr ∂vθ ∂vz + δθ × δr + vr δθ + δθ − vθ δr + δz r ∂θ ∂θ ∂θ     ∂vθ ∂vz ∂vr ∂vθ = δz − δθ + δθ − δ ∂r ∂r ∂z ∂z r   1 ∂vr ∂vθ + − δ + vθ δz + δ r ∂θ z ∂θ r       1 ∂vz ∂vθ ∂vz ∂vr 1 ∂(rvθ ) ∂vr = δr + − − δθ + + δz − r ∂θ ∂z ∂r ∂z r ∂r ∂θ (4.74)

186

Vector and Tensor Algebra and Calculus Table 4.5. Relationship between rectangular and cylindrical coordinates Rectangular

Cylindrical

1

Unit Vectors δx δy δz

= = =

cos θδr − sin θδθ sin θδr + cos θδθ δz

2

δr δθ δz

= = =

cos θδx + sin θδy − sin θδx + cos θδy δz

Vector Components v = vx δx + vy δy + vzδz vx vy vz

= = =

v = vr δr + vθ δθ + vzδz

vr cos θ − vθ sin θ vr sin θ + vθ cos θ vz

3

vr vθ vz

= = =

vx cos θ + vy sin θ −vx sin θ + vy cos θ vz

Partial Differential Operators ∂ ∂x ∂ ∂y ∂ ∂z

= = =

  ∂ sin θ − ∂r  r  ∂ cos θ sin θ + ∂r r ∂ ∂z

cos θ

4

∂ ∂θ ∂ ∂θ

∂ ∂r ∂ ∂θ ∂ ∂z

= = =

∂ ∂ + sin θ ∂x ∂y ∂ ∂ −r sin θ + r cos θ ∂x ∂y ∂ ∂z cos θ

Gradient Operators ∇ = δx

∂ ∂ ∂ + δy + δz ∂x ∂y ∂z

5

∇ = δr

∂ 1 ∂ ∂ + δθ + δz ∂r r ∂θ ∂z

Derivatives of Unit Vectors ∂δk =0 ∂m for k, m = x, y, z

∂δr = δθ ; ∂θ

∂δθ = −δr ∂θ

zero for all other cases

A fluid flowing through a pipe of radius R attains a steady-state velocity of a Poiselle flow given by   r2 v (r, θ, z) = vmax 1 − 2 δz R

EXAMPLE 4.15.

which is a paraboloid-shaped velocity profile that is symmetric along the z-axis (flow direction), as shown in Figure 4.20. The divergence of v is zero; that is, it is a solenoidal field. The curl field is given by r ∇ × v = 2vmax 2 δθ R The curl field is also given in Figure 4.20, which shows the curl field varying linearly with increasing radius. This means that very small particles near r = R would experience the maximum curl due to the velocity field around it and zero curl at the center.

4.8 Curvilinear Coordinate System: Cylindrical and Spherical

1

1

z 0.5

z 0.5

0

0

1

1 0.5

1

1 0.5

0.5 0

y

−1

0 −0.5

−0.5 −1

0.5 0

0 −0.5

187

y

x

−0.5 −1

−1

x

Figure 4.20. The Poiselle velocity field at z = 0 (left plot) and the corresponding curl field (right plot).

4.8.2 Spherical Coordinate System The spherical coordinate system is a coordinate system as shown in Figure 4.21 and defined by the following invertible transformations: ! r = x2 + y2 + z2 x = r sin θ cos φ ! −1 ↔ y = r sin θ sin φ (4.75) θ = tan ( x2 + y2 /z) φ

=

tan−1 (y/x)

z

=

r cos θ

A summary of important relationships between the rectangular and spherical coordinate systems is given in Table 4.6. In the table, as well as in several places in this chapter, we use the shorthand notation for sines and cosines (i.e., sθ = sin θ, cθ = cos θ, sφ = sin φ, and cφ = cos φ). The derivation of the items in this table can be done via geometric arguments. Details of these derivations, aided by matrix methods, are given in Section D.3.

Figure 4.21. The spherical coordinate system.

188

Vector and Tensor Algebra and Calculus Table 4.6. Relationship between rectangular and spherical coordinates Rectangular

Spherical

1

Unit Vectors δx δy δz

= = =

sθ cφ δr + cθ cφ δθ − sφ δφ sθ sφ δr + cθ sφ δθ + cφ δφ cθ δr − sθ δθ

2

δr δθ δφ

= = =

sθ cφ δx + sθ sφ δy + cθ δz cθ cφ δx + cθ sφ δy − sθ δz − sφ δx + cφ δy

Vector Components v = vx δx + vy δy + vzδz

v = vr δr + vθ δθ + vφ δφ

vx vy vz

vr vθ vφ

= = =

sθ cφ vr + cθ cφ vθ − sφ vφ sθ sφ vr + cθ sφ vθ + cφ vφ cθ vr − sθ vθ

3

= = =

sθ cφ vx + sθ sφ vy + cθ vz cθ cφ vx + cθ sφ vy − sθ vz − sφ vx + cφ vy

Partial Differential Operators ∂ ∂x

=

∂ ∂y

=

∂ ∂z

=

c c  ∂ ∂ θ φ + ∂r ∂θ   r sφ ∂ − rsθ ∂φ c s  ∂ ∂ θ φ sθ sφ + ∂r ∂θ   r cφ ∂ + rsθ ∂φ s  ∂ ∂ θ cθ − ∂r r ∂θ

sθ c φ

4

∂ ∂r

=

∂ ∂θ

=

∂ ∂φ

=

∂ ∂ + sθ sφ ∂x ∂y ∂ + cθ ∂z ∂ ∂ r cθ cφ + r cθ sφ ∂x ∂y ∂ − r sθ ∂z ∂ ∂ −r sθ sφ + r sθ cφ ∂x ∂y sθ c φ

Gradient Operators ∇ = δx

∂ ∂ ∂ + δy + δz ∂x ∂y ∂z

5

∇ = δr

∂ 1 ∂ 1 ∂ + δθ + δφ ∂r r ∂θ rsθ ∂φ

Derivatives of Unit Vectors ∂δr ∂δθ ∂δr = δθ ; = −δr ; = δφ sθ ∂θ ∂θ ∂φ ∂δφ ∂δθ = δφ cθ ; = −δr sθ − δθ cθ ∂φ ∂φ

∂δk =0 ∂m for k, m = x, y, z

zero for all other cases

Using these operators, we can find the divergence and curl of a vector field to be     ∂vr 1 ∂vθ vr 1 ∂vφ vθ cos θ ∇ ·v= + + 2 + + (4.76) ∂r r ∂θ r sin θ ∂φ r r sin θ and ∇ ×v



=

   1 ∂vφ 1 ∂vθ vφ cos θ 1 ∂vr ∂vφ vφ − + δr + − − δθ r ∂θ r sin θ ∂φ r sin θ r sin θ ∂φ ∂r r   ∂vθ 1 ∂vr vθ + − + (4.77) δφ ∂r r ∂θ r

4.9 Orthogonal Curvilinear Coordinates

189

EXAMPLE 4.16. Let the v be a vector field that is a function of the position vector r, away from the origin, given by α r v= (r · r)n/2

Instead of using the rectangular coordinate system, where r = xδx + yδy + zδz, the spherical coordinates become more convenient because r = rδr , which results in α v = n−1 δr r With vθ = vφ = 0 and vr = vr (r), the curl can be evaluated using (4.77) to yield a zero vector ∇ ×v=0 that is, v is irrotational. Conversely, the divergence of v can be evaluated using (4.76) to yield ∂vr vr 3−n +2 =α n ∂r r r and v becomes solenoidal for n = 3.

∇ ·v=

4.9 Orthogonal Curvilinear Coordinates 4.9.1 Definitions and Notations We now generalize the results of cylindrical and spherical coordinate systems to other orthogonal curvilinear coordinates. Let the new coordinates be given by (a, b, c), that is, x y z

= = =

x (a, b, c) y (a, b, c) z (a, b, c)

↔

a b c

= = =

a (x, y, z) b (x, y, z) c (x, y, z)

(4.78)

Then, according to the implicit value theorem, the transformation between the two coordinates will exist if the Jacobian matrix given by ⎛

J (x,y,z)→(a,b,c)

is nonsingular.

∂x ⎜ ∂a ⎜ ⎜ ⎜ ∂y ⎜ =⎜ ⎜ ∂a ⎜ ⎜ ⎜ ∂z ⎝ ∂a

∂x ∂b ∂y ∂b ∂z ∂b

⎞ ∂x ∂c ⎟ ⎟ ⎟ ∂y ⎟ ⎟ ⎟ ∂c ⎟ ⎟ ⎟ ∂z ⎟ ⎠ ∂c

(4.79)

190

Vector and Tensor Algebra and Calculus

Figure 4.22. The a-coordinate curve and the a-coordinate surface.

Definition 4.6. Let the Jacobian in (4.79) be nonsingular for the new coordinate (a, b, c) given in (4.78). The a-coordinate curve (or a-curve) is the locus of points where b and c are fixed. The a-coordinate surface (or a-surface) is the surface defined at a fixed a.

Figure 4.22 shows the a-coordinate curve together with the a-coordinate surface. Using (4.78), the a-curve at b = b0 and c = c0 is given by a set of points described by the position vectors r (a, b0 , c0 ) = x (a, b0 , c0 ) δx + y (a, b0 , c0 ) δy + z (a, b0 , c0 ) δz whereas the a-surface at a = a0 is described by the scalar function a0 = a (x, y, z) Definition 4.7. Let p be a point in the 3D space. A vector νa (p) that is tangent to the a-coordinate curve at p is called the a-base vector and defined as     ∂k  ∂  r  = νa (p) = δk  (4.80) ∂a p ∂a  k=x,y,z p

A vector  νa (p) that is normal to the a-coordinate surface at p is called the areciprocal base vector, or a-dual base vector, and defined as       ∂a    νa (p) = ∇ a (x, y, z)  = δk  (4.81) p ∂k  k=x,y,z p

The a-base vector νa and a-reciprocal base vector νa at P are shown in Figure 4.23. Note that νa and  νa do not necessarily point in the same direction. Furthermore, neither νa nor  νa will necessarily have unit length. The base vectors and reciprocal base vectors can be normalized easily by dividing by their respective norms. We denote the unit base vectors by δa , and we denote the unit reciprocal base vectors by  δa , where δa =

νa νa

and

 ν  δa = a  νa

A similar set of definitions apply to the b-coordinate curves, b-coordinate surfaces, b-base vectors, and b-reciprocal base vectors, as well as to c-coordinate curves, ccoordinate surfaces, c-base vectors, and c-reciprocal base vectors.

4.9 Orthogonal Curvilinear Coordinates

191

Figure 4.23. The a-base vector and the a-reciprocal base vector.

Based on Definitions 4.6 and 4.7, we have the following results and observations: 1. Vector  νa is orthogonal to νb and νc ,  νa · νb

=

∂a ∂x ∂a ∂z ∂a ∂z ∂a + + = =0 ∂x ∂b ∂y ∂b ∂z ∂b ∂b

 νa · νc

=

∂a ∂x ∂a ∂z ∂a ∂z ∂a + + = =0 ∂x ∂c ∂y ∂c ∂z ∂c ∂c

(4.82)

because a is independent of b and c. Similarly, νb is orthogonal to νa and νc , and  νc is orthogonal to νa and νb. 2. The dot product of  νa and νa is unity, that is,  νa · νa =

∂a ∂x ∂a ∂z ∂a ∂z ∂a + + = =1 ∂x ∂a ∂y ∂a ∂z ∂a ∂a

(4.83)

Similarly,  νb · νb = 1 and  νc · νc = 1. 3. The set (νa , νb, νc ) forms a linearly independent set of vectors that span the 3D space. Thus any vector v can be represented by a linear combination of the base vectors, that is, v = v¯ a νa + v¯ bνb + v¯ c νc Likewise, the reciprocal base vectors ( νa , νb, νc ) form another linearly independent set of basis vectors. However, they are used more as the basis for the gradient operator ∇. To see this, we start with the gradient of a scalar field in rectangular coordinates, ∇ψ

=



δk

k=x,y,z

=

k=x,y,z

m=a,b,c

 ∂ψ   ∂ψ ∂m δk = ∇m ∂m ∂k ∂m

m=a,b,c

=

  ∂ψ ∂m ∂ψ = δk ∂k ∂m ∂k

k=x,y,z

m=a,b,c

  ∂ ∂ ∂  νa + νb + νc ψ ∂a ∂b ∂c

or ∇ = νa

∂ ∂ ∂ + νb + νc ∂a ∂b ∂c

(4.84)

192

Vector and Tensor Algebra and Calculus

4.9.2 Orthogonal Coordinate Systems We now consider the special case of orthogonal coordinate systems, in which the base vectors and unit base vectors form an orthogonal set such that δ a = δb × δc

δb = δc × δa

δc = δa × δb

(4.85)

Because  νa is normal to the a-surface, the orthogonality of the base vectors means that  νa and νa are pointing in the same direction. After normalizing the base vectors and reciprocal base vectors, we have  δa = δa

;

 δb = δb

 δc = δc

;

(4.86)

In addition, we can use the fact that νa · νa = 1, νb · νb = 1, and νc · νc = 1 to show that  νa =

1 νa

 νb =

;

1 νb

 νc =

;

1 νc

(4.87)

We call the norms of the base vectors scaling factors, denoted by αa , αb and αc , αa =

νa

=

∂r ∂a

;

αb =

νb

=

∂r ∂b

;

αc =

νc

=

∂r ∂c

(4.88)

This means that for orthogonal base vectors, the gradient operator can also be written in terms of the base vectors or unit base vectors as follows: ∇

= = =

∂ ∂ ∂ + νb + νc ∂a ∂b ∂c ∂ ∂ ∂  νa δˆ a νb δˆ b +  +  νc δˆ c ∂a ∂b ∂c 1 ∂ 1 ∂ 1 ∂ δ + δ + δ αa a ∂a αb b ∂b αc c ∂c

 νa

(4.89)

Consider the parabolic coordinate system (a, b, θ) defined by the following equations:

EXAMPLE 4.17.

 1 2 a − b2 2 which is a valid coordinate system in a domain where the following Jacobian is nonsingular: ⎛ ⎞ b cos θ a cos θ −ab sin θ ⎜ ⎟ J rect→parabolic = ⎝ b sin θ a sin θ ab cos θ ⎠ a −b 0

x = ab cos θ

;

y = ab sin θ

;

z=

Solving for a, we have x2 + y2 b2

= =

a2 b2 a2 − 2z

→

a=

√ z+ r

4.9 Orthogonal Curvilinear Coordinates

193

! where r = x2 + y2 + z2 . At fixed values of a, this gives the a-coordinate surface. Likewise, for the b-coordinate surface, we have x2 + y2 a2

= =

a2 b2 b2 + 2z

→

b=

√ −z + r

  whereas for the θ-coordinate surface, we have θ = Atan xy , which is the same as the cylindrical coordinate θ, that is, a plane containing the z-axis. Let r be the position vector. Then the base vectors are given by νa

=

νb

=

νθ

=

∂r = b cos θδx + b sin θδy + aδz ∂a ∂r = a cos θδx + a sin θδy − bδz ∂b ∂r = −ab sin θδx + ab cos θδy ∂θ

This is an orthogonal coordinate system because νa · νa = a2 + b2 = νb · νb

;

νθ · νθ = a2 b2

and the other dot products are zero. The scaling factors are given by the square root of these dot products, that is, ! αa = αb = a2 + b2 ; αθ = ab which then yield the gradient operator in the parabolic coordinate system to be ∇=√

1 a2 + b2

δa

∂ 1 ∂ 1 ∂ +√ δb + δθ ∂a ∂b ab ∂θ a2 + b2

Based on (4.89), we can obtain some of the important differential identities in terms orthogonal coordinates (a, b, c) and the unit base vectors: 1. Gradient ∇ψ =

1 ∂ψ 1 ∂ψ 1 ∂ψ δ + δ + δ αa a ∂a αb b ∂b αc c ∂c

(4.90)

2. Divergence ∇ ·w=

1 αa αbαc



∂ ∂ ∂ αbαc wa + αc αa wb + αa αbwc ∂a ∂b ∂c

 (4.91)

3. Curl ⎛ ⎜ ⎜ 1 ∇ ×w= det ⎜ ⎜ αa αbαc ⎝

αa δa

αbδb

αc δc

∂ ∂a

∂ ∂b

∂ ∂c

αa wa

αbwb

αc wc

⎞ ⎟ ⎟ ⎟ ⎟ ⎠

(4.92)

194

Vector and Tensor Algebra and Calculus

4. Laplacian of Scalar Fields ⎡

∇ 2ψ =

1 ⎢ ∂ ⎣ αa αbαc ∂a



αbαc ∂ψ αa ∂a

 +

∂ ∂b



αa αc ∂ψ αb ∂b

 +

∂ ∂c



αa αb ∂ψ αc ∂c



⎤ ⎥ ⎦ (4.93)

5. Gradient-Vector Dyad

∇w =





k=a,b,c m=a,b,c

 1 ∂wk δm δk + αm ∂m

 wk ∂δ k δ αm m ∂m

(4.94)

k=a,b,c m=a,b,c

The proofs of (4.90) to (4.94) are given in the appendix as Section D.1. Applying these results to cylindrical and spherical coordinates, we obtain an alternative set of combined formulas given by the following: Gradient of Scalar Fields:

∇ψ

=

 1 ∂ψ δ αk ∂k k k

Partial Derivative of Vector Fields:

∂ v ∂m

=

Divergence of Vector Fields:

∇ ·v

=

  ∂vk  ∂m

k

2   1 ∂vk  + + D (v) αk ∂k k

⎛

Curl of Vector Fields:

+ 2 δk + pm (v)

∇ ×v

=

δa

⎜ ⎜ 1 ∂ ⎜ det ⎜ ⎜ αa ∂a ⎝

δb

δc

1 ∂ αb ∂b

1 ∂ αc ∂c

⎞ ⎟ + 2 ⎟ ⎟ ⎟ + q (v) ⎟ ⎠

vb vc va where (a, b, c) = (x, y, z) , (r, θ, z) or (r, θ, φ)

( Laplacian of Scalar Fields:

2

∇ ψ

=

Gradient-vector dyads:

∇v

=

2  1 ∂2ψ + + L (ψ) α2k ∂k2 k + 2    1 ∂vk  δ δ + T (v) αm ∂m m k m

)

(4.95)

k

The values of parameters αk , q, D, L, p, and T are given in Table 4.7 for both the cylindrical and spherical coordinates. Note that the preceding equations also apply to rectangular coordinates by setting αx = αy = αz = 1 and disregarding all the terms with curly brackets.

4.9 Orthogonal Curvilinear Coordinates Table 4.7. Parameters for (4.95) for cylindrical and spherical coordinates Cylindrical

Spherical

α

αr = 1, αθ = r, αz = 1

αr = 1, αθ = r, αφ = rsθ

C

q (v) =

vθ δ r z

q (v) =

D

D (v) =

vr r

D (v) = 2

L

L (ψ) =

1 ∂ψ r ∂r

L (ψ) =

P

pθ (v) = −vθ δr + vr δθ

T

vθ vr T (v) = − δθ δr + δθ δθ r r

vφ cθ vφ vθ δ − δθ + δφ rsθ r r r vr vθ cθ + r rsθ

2 ∂ψ cθ ∂ψ + 2 r ∂r r sθ ∂θ

pθ (v)

=

−vθ δr + vr δθ

pφ (v)

=

−vφ sθ δr − vφ cθ δθ + (vr sθ + vθ cθ ) δφ

T (v)

=

−

vθ vr δ δ + δθ δθ r θ r r   vφ vφ cθ vr vθ cθ − δφ δr − δφ δθ + + δφ δφ r rsθ r rsθ

Note: Absent items are considered zero, e.g. p r = 0. (Arranged in alphabetical order.)

Let us show that the formulas given in (4.92) and (4.95) for the curl of a vector field in spherical coordinates are the same as that given by (4.77). Applying (4.92) for the spherical coordinates, with (a, b, c) = (r, θ, φ) and (αa , αb, αc ) = (1, r, r sin θ), gives ⎛ ⎞ δr rδθ r sin θδφ ⎜ ⎟ 1 ∂ ∂ ∂ ⎜ ⎟ ∇ ×v = det ⎜ ⎟ 2 ⎝ ⎠ r sin θ ∂r ∂θ ∂φ EXAMPLE 4.18.

r sin θvφ    1 ∂(r sin θvφ ) ∂(rvθ ) 1 ∂vr ∂(r sin θvφ ) δ − + δ − r2 sin θ r ∂θ ∂φ r sin θ θ ∂φ ∂r   1 ∂(rvθ ) ∂vr + δφ − r ∂r ∂θ vr



=

rvθ

which can be shown to be the same as (4.77). Alternatively, let us check (4.95) for the curl in spherical coordinates, ⎛ ⎞ δr δθ δφ   ⎜ ∂ vφ cos θ vφ vθ 1 ∂ 1 ∂ ⎟ ⎜ ⎟ ∇ × v = det ⎜ δ r − δ θ + δφ ⎟+ ⎝ ∂r ⎠ r sin θ r r r ∂θ r sin θ ∂φ 

=

vr

vθ



vφ

    1 ∂vφ 1 ∂vθ 1 ∂vr ∂vφ ∂vθ 1 ∂vr δr − + δθ − + δφ − r ∂θ r sin θ ∂φ r sin θ ∂φ ∂r ∂r r ∂θ   vφ cos θ vφ vθ + δ r − δ θ + δφ r sin θ r r

which can also be shown to yield (4.77).

195

196

Vector and Tensor Algebra and Calculus 4.10 EXERCISES

E4.1. Verify all the properties in Table 4.2 using the following values: α = 1.5 u = δx + 2δy + 2δz w = 2δy + δz

β = −2 v = δx + δy + 2δz y = δx + 4δz

; ; ;

E4.2. Prove or disprove: The dot product and cross product can be interchanged in a triple product, that is, u · (v × w) = (u × v) · w E4.3. Consider three non-collinear points A = (ax , ay , az), B = (bx , by , bz), and C = (cx , cy , cz). The minimum distance d from point C to the line passing through A and B is given by the following formula d= where u

=

v

=

u×v v

(4.96)

  (cx − ax ) δx + cy − ay δy + (cz − az) δz   (bx − ax ) δx + by − ay δy + (bz − az) δz

Show that the same value results if the roles of A and B had been interchanged. E4.4. Prove or disprove the following identities: 1. (a × b) · c = a · (b × c) 2. (a × b) · (c × d) = (a · c) (b · d) − (a · d) (b · c) 3. (a × b) × (c × d) = (a · (c × d)) b − (b · (c × d)) a Also, verify the proven identities using a = 2δx + 3δy − δz

;

b = 3δx + 2δz

c = 4δx − 2δy + 2δz

;

d = −2δy + 3δz

E4.5. Determine a formula for the volume of the triangular pyramid whose vertices are (xi , yi , zi ), i = 1, 2, 3, 4. Under what conditions will the volume be zero? E4.6. A triangle in 3D by three non-collinear vertices given by  space  is determined     a = ax , ay , az , b = bx , by , bz , and c = cx , cy , cz . Let the oriented triangular area, denoted by A (a, b, c), be the vector defined by 1 A (a, b, c) = (b − a) × (c − a) (4.97) 2 where a, b, and c are the position vectors based on the coordinates of a, b, and c. 1. Show that the area of the triangle abc is given by A (a, b, c) . Use this to find the area formed by a = (1, 1, 1), b = (1, 0, 1) and c = (1, −1, 1). 2. Show that an alternative formula is given by 1 A (a, b, c) = (a × b + b × c + c × a) 2 thus also show that A (a, b, c) = A (b, c, a) = A (c, a, b) 3. Consider the tetrahedron described by four vertices a, b, c, and d as shown in Figure 4.24.

4.10 Exercises

197 c z

Figure 4.24. A tetrahedron described by four points: a, b, c, and d.

y

a d

x

Based on the right-hand rule, the outward normal vectors will be based on the sequences (within a cyclic permutation) abc, bdc, cda, and dba. Show that the vector sum of the oriented areas will be a zero vector, that is, A (a, b, c) + A (b, d, c) + A (c, d, a) + A (d, b, a) = 0 Verify this fact using a = (1, 2, 1), b = (2, 1, 1), c = (2, 2, 4) and d = (3, 3, 2). E4.7. Consider the following symmetric stress tensor T and a unit normal n given by ⎛ ⎞ ⎛ ⎞ 2 3 1 1 1 T =⎝ 3 1 1 ⎠ ; n= √ ⎝ 1 ⎠ 3 1 1 2 1 1. Find the normal stress vector and shear stress vector that correspond to the surface determined by the unit normal vector n. 2. Show that, in general, given stress tensor T and unit normal n, the normal stress vector τnormal and shear stress vector τ shear are given by     τnormal = nn · T ·n     τ shear = I −nn · T ·n where I is the identity tensor. Verify these formulas by comparing them with the results you found previously. 3. Find the unit vectors, v1 , v2 , and v3 along the principal axes of the stress tensor T . Show that the normal stress along v1 is equal to v1 multiplied by the corresponding eigenvalue of T while the shear stress is zero. Show that the same is true for v2 and v3 . E4.8. A set of 3D vectors, u, v, and w, spans the whole 3D space. Another related set of three vectors is called the reciprocal vectors. A vector  u is the reciprocal vector of u if: (1) it is orthogonal to v and w, and (2)  u · u = 1. Find the formulas for the reciprocal vectors,  u,  v, and  w. (Hint: Use cross products and triple products.) Note: The reciprocal vectors are used to define tensors in case the basis vectors are not orthogonal. E4.9. For the curve C given by x(t) = 3 cos (t), y = sin (t), and z = t/(1 + 2t), evaluate the following at t = 5: 1. Velocity v and acceleration a  , and binormal unit 2. The tangent unit vector t, normal unit vector n  vector b 3. The tangential and normal components of acceleration a 4. The curvature κ and torsion τ

b

198

Vector and Tensor Algebra and Calculus

5. Verify the following equations known as the Frenet formulas at the given point: dt = κ n ds d n  = −κt + τb ds  db = −τ n ds Hint: For a vector c, use the relationship dc dc/dt = ds s˙ E4.10. A solid top is shown in Figure 4.25 and has its surface described by the following equations: x

=

sin (θ) cos (φ)

y

=

sin (θ) sin (φ)

z

=

1.5 cos (θ/2)

for 0 ≤ θ ≤ π and 0 ≤ φ ≤ 2π.

1.5

z

1

Figure 4.25. Solid surface given for problem E4.10.

0.5

0.5

0.5

0 y

-0.5

-0.5

0

x

Obtain the unit normal pointing outward at the point corresponding to θ = φ = π/4. Verify by plotting the unit normal along with the figure. (What about the unit normal at θ = π ?) E4.11. Householder Tensor. We could develop the Householder operator (cf. (3.6)) for physical vectors. This tensor is supposed to reflect any given vector v across a chosen plane P. Let us denote this tensor as Hw , where w is a vector that is normal to the chosen plane. Also, define vectors a, b, and u by those shown in Figure 4.26. 1. Obtain D, the dyad that would project v onto w and pointing along w, that is,   a=D v 2.

From Figure 4.26, we see that the desired vector u is given by u = b + (−a)

4.10 Exercises

199

Figure 4.26. Householder operation on v, based on w.

where b = v − a. Using these results, obtain the Householder tensor that takes vector v and yields vector w. E4.12. Metric Tensors. Consider the general case in which the invertible transformation between the Cartesian coordinate system and another coordinate systems (a, b, c) are given by: a b c

= = =

a (x, y, z) b (x, y, z) c (x, y, z)

x y z

⇐⇒

= = =

x (a, b, c) y (a, b, c) z (a, b, c)

Let r be the position vector. A Riemann space is a space equipped with a metric given by ds2 = dr · dr dr

1.

=

∂r ∂r ∂r dx + dy + dz ∂x ∂y ∂z

=

∂r ∂r ∂r da + db + dc ∂a ∂b ∂c

For the Cartesian coordinates, we know that ds2 = dx2 + dy2 + dz2 . Show that the matrix G = J T J , where J is the Jacobian matrix, ⎛ ⎞ ∂x ∂x ∂x ⎜ ∂a ∂b ∂c ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ∂y ∂y ∂y ⎟ ⎜ ⎟ ⎟ J =⎜ ⎜ ∂a ∂b ∂c ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ∂z ∂z ∂z ⎟ ⎝ ⎠ ∂a ∂b ∂c will satisfy the equation, ds2 =

2.



da

db

dc



⎛

⎞ da G ⎝ db ⎠ dc

The elements of matrix G contains the components of G, the metric tensor (also known as the fundamental tensor). A coordinate system will be orthogonal if G is diagonal. Show that the cylindrical coordinate system, that is, with (a, b, c) = (r, θ, z), and the spherical coordinate system, that is, with (a, b, c) = (r, θ, φ), are two examples of orthogonal coordinates systems.

200

Vector and Tensor Algebra and Calculus

3.

The unit vectors of the new coordinate system can be obtained as ∂r/∂a ∂r/∂b ∂r/∂c δa = δb = δc = ∂r/∂a ∂r/∂b ∂r/∂c For the special case of orthogonal coordinate systems, the formula for ds2 will result in the relationship between the unit vectors of the rectangular coordinate system and the unit vectors of the other orthogonal system: ⎛ ⎞ ⎛ ⎞ δx δa ⎝ δy ⎠ =  ⎝ δb ⎠ δc δz where

  = J diag

4.

1 √ G11

1 √ G22

1 √ G33



because G = J T J is diagonal for orthogonal systems. Verify that the orthogonal matrices Rr→c and Rr→s given in (D.4) and (D.11), respectively, can be obtained by evaluating  for each case. For orthogonal systems, obtain the diagonal matrix  such that ⎛ ⎛ ⎞ ⎞ ∂ ∂ ⎜ ∂a ⎟ ⎜ ∂x ⎟ ⎜ ⎜ ⎟ ⎟ ⎜ ∂ ⎟ ⎜ ∂ ⎟ ⎜ ⎜ ⎟ ⎟ T ⎜ ⎜ ⎟ ⎟ ⎜ ∂b ⎟ =  ⎜ ∂y ⎟ ⎜ ⎜ ⎟ ⎟ ⎜ ∂ ⎟ ⎜ ∂ ⎟ ⎝ ⎝ ⎠ ⎠ ∂c ∂z Verify your results by applying them to both the cylindrical and spherical coordinate systems. Note that at this point, the basic ingredients, that is,  and , are now available to find ∇ and ∂δk /∂m, for k = a, b, c and m = a, b, c.

E4.13. Gradient Ascent Method. Consider the scalar function  1  2 ψ (x, y) = 8x + 8x − 4xy + 16y + 5y2 + 38 18 1. Obtain the gradient field of ψ (x, y). (0) −2 and 2. Let x(0) =  y = 0. Find the equation of the line that passes  (0) (0) and oriented along the gradient at that point. Call this through x , y line L1 . 3. Plot the of the directional  magnitude   derivative of ψ along the line L1 from x(0) , y(0) up to a point x(1) , y(1) where the directional derivative is zero.    (1) (1)  4. Repeat the process x , y  instead of x(0) , y(0) , and iterate one  with more time to find x(2) , y(2) and x(3) , y(3) . 5. Obtain a contour plot of ψ (x, y) and overlay this plot with the path obtained by the gradient ascent   approach,   that is,  the segments  connect ing, in sequence: x(0) , y(0) , x(1) , y(1) , x(2) , y(2) and x(3) , y(3) .13 13

Instead of ending each iteration at the point where the directional derivative is zero, which could mean solving nonlinear equations, one could simply take a path along the gradient line and end at

4.10 Exercises

201

E4.14. Prove the following identity ∇ · (vw) = (∇ · v) w + v · ∇w

(4.98)

E4.15. Prove the following identity: ∇ 2 (∇ · v) = ∇ · ∇ 2 v

(4.99)

E4.16. Prove the following identity:     ∇ · T · v = v · ∇ · T + T : ∇v where ∇ ·T =

 ∂T ij δ ∂i j i,j =x,y,z

and

T : ∇v =

(4.100) 

i,j =x,y,z

T ij

∂v j ∂i

E4.17. Let ψ and ω be scalar fields. Using the identities given in (4.57) and (4.60), show that ∇ · (∇ψ × ∇ω) = ∇ · (∇ω × ∇ψ) = 0 that is, ∇ψ × ∇ω is solenoidal. E4.18. Using the identity (4.98) and the continuity equation (4.64), show that following is true ∂ρv Dv = ρ + ∇ · ρv v (4.101) Dt ∂t where D/Dt is the substantial time derivative defined in (4.65). E4.19. The stress tensor T of a fluid that follows Newton’s law of viscosity is given by    2 T = −μ ∇v + (∇v)T + μ − κ (∇ · v) I (4.102) 3 where μ, κ, and I are the viscosity, dilatational viscosity, and identity tensor, respectively. 1. Define the tensor (∇v)T by (∇v)T =

 j,k=x,y,z

∂v j δδ ∂k j k

where v is the velocity vector field. Then show that ∇ · (∇v)T = ∇ (∇ · v) 2. For the case where density ρ is constant, the equation of continuity given in (4.66) becomes ∇ · v = 0. Using this assumption, show that the divergence of the stress tensor given in (4.102) becomes ∇ · T = −μ∇ 2 v

(4.103)

This means that equation of motion given in (5.26) reduces to the equation known as the Navier-Stokes equation of motion, Dv = −∇ p + μ∇ 2 v + ρg ρ Dt where p and g are the pressure and gravitational acceleration, respectively. a fixed ratio of the length of the gradient. Using a small ratio has a better chance of not missing the optimum along the path. However, a small ratio slows down the convergence.

202

Vector and Tensor Algebra and Calculus

E4.20. A material following the Euler’s equation of motion (e.g., inviscid flow) is described in (5.28), which can be rewritten to be ∂v + v · ∇v = −∇ (Q + ) (4.104) ∂t where Q is a potential body force field per unit mass and  is a scalar field whose gradient is defined by ∇ = (∇ p ) /ρ, assuming ρ is a function of pressure p only. 1. Using the identities given in Section 4.7.6, show that ∂ω + v · ∇ω = ω · ∇v − ω (∇ · v) ∂t where ω = ∇ × v is the vorticity of v. Or, using the substantial time derivative operator defined in (4.65), Dω = ω · ∇v − ω (∇ · v) (4.105) Dt 2. By dividing (4.105) by ρ and then using the equation of continuity given in (4.66), show that   D 1 1 ω = ω · ∇v (4.106) Dt ρ ρ This equation is known as the Helmholtz vorticity equation for inviscid fluids with ρ = ρ (p ). E4.21. Obtain h(v) = v · (∇ × v) in rectangular coordinates. When h(v) = 0, then v is not perpendicular to (∇ × v). Show that this is the case for   v (x, y, z) = xyz δx + δy + δz for x = y and z = 0. For the special case where v = vx (x, y) δx + vy (x, y) δy + 0δz, what is h (v)? What can one say about h (v) when v = ∇ψ (x, y, z), where ψ is any smooth scalar function ? E4.22. Obtain ∇ 2 v in cylindrical coordinates. E4.23. Prove the following identity: (v · ∇) v =

 1 2 ∇ v − v × (∇ × v) 2

(Hint: Use identity (4.59).) E4.24. Prove or disprove the following statement:     2 = 2∇ψ · ∇ ∇ψ ∇ ∇ψ E4.25. The temperature distribution for a flat rectangular plate is given by T (x, y) = 1 − eg(x,y) where, g (x, y) = −4



(x − 0.5)2 + (y + 0.5)2



for −1 ≤ x ≤ 1 and −1 ≤ y ≤ 1. 1. Calculate ∇T (x, y). 2. Obtain a contour plot of the temperature distribution and overlay it with the gradient vector field plot. 3. Evaluate ∇ 2 T and its gradient.

4.10 Exercises

4. 5.

Obtain another contour plot, this time of the Laplacian of T . Overlay it with the gradient field of the Laplacian. What are the differences between the two plots?

E4.26. Find the relationship between the unit vectors of the cylindrical coordinate system with the unit vectors of the spherical coordinate system, that is, determine Rc→s such that ⎛ ⎞ ⎛ ⎞ δr δr ⎝ δθ ⎠ = Rc→s ⎝ δθ ⎠ δφ δz (Hint: Use the matrices found in Sections D.2 and D.3.) (Note that we use  r and  θ for the spherical coordinates to distinguish them from the r and θ variables, respectively, used for the cylindrical coordinates.) E4.27. For a tensor T , find the relationship between the components in the rectangular coordinates and the components in the spherical coordinates, for example, T rr = T rr (T xx , . . . , T zz), and so forth. E4.28. Show that (4.93) is equal to the formula for the Laplacian of a scalar field ψ given in (4.95) for the case of spherical coordinates. E4.29. Given the vector field in the spherical coordinates,   1 1 v (r, θ, φ) = 2 4 − 3 δr r r and a constant vector field w in rectangular coordinates w = 2δx + 4δy Evaluate (w · ∇v) at r = 1.2 and θ = φ = π/4 and give the results in spherical coordinates, as well as in rectangular coordinates. E4.30. For the parabolic coordinate system described in Example 4.17, obtain the divergence and curl of a vector field v in this coordinate system. Also, obtain the reciprocal base vectors  νa ,  νb, and  νθ and show that νa · νa = 1, νb · νb = 1 and νθ · νθ = 1.

203

5

Vector Integral Theorems

In this chapter, we discuss the major integral theorems that are used to develop physical laws based on integrals of vector differential operations. The general theorems include the divergence theorem, the Stokes’ theorem, and various lemmas such as the Green’s lemma. The divergence theorem is a very powerful tool in the development of several physical laws, especially those that involve conservation of physical properties. It connects volume integrals with surface integrals of fluxes of the property under consideration. In addition, the divergence theorem is also key to yielding several other integral theorems, including the Green’s identities, some of which are used extensively in the development of finite element methods. Stokes’ theorem involves surface integrals and contour integrals. In particular, it relates curls of velocity fields with circulation integrals. In addition to its usefulness in developing physical laws, Stokes’ theorem also offers a key criteria for path independence of line integrals inside a given region that can be determined to be simply connected. We discuss how to determine whether the regions are simply connected in Section 5.3. In Section 5.5, we discuss the Leibnitz theorems involving the derivative of volume integrals in both 1D and 3D space with respect to a parameter α in which the boundaries and integrands are dependent on the same parameter α. These are important when dealing with time-dependent volume integrals. In applying the various integral theorems, some computations may be necessary. To this end, we have included extensive discussions of the various types of integrals in Sections E.1 through E.3 (i.e., line integrals, surface integrals, and volume integrals). In these sections, we include the details for computation and some examples to appreciate the implications and evaluations of the various integral theorems useful for the integral theorems that are covered in this chapter. In Section 5.4, we discuss two very important applications of the integral theorems. These are the development of conservation laws and the development of Maxwell equations for electricity and magnetism. Both these applications expose the power of vector differential operators for the development of the partial differential equations that model several physical systems. We do not cover the actual solution of partial differential equations in this chapter. The solution of these partial differential equations can be very complex and is often difficult to obtain in closed analytical form except for specific cases. 204

5.1 Green’s Lemma

205

Instead, their solutions are dealt with in later chapters, starting with Chapter 6 and succeeding chapters for handling problems that can be reduced to ordinary differential equations; Chapters 10 through 14 handle solution of certain classes of partial differential equations, containing both analytical and numerical approaches.

5.1 Green’s Lemma We begin with one of the basic formulas that consists of the relationship between a surface integral on surface S and its boundary C. Green’s Lemma. Let F (u, v) and G(u, v) be differentiable functions in the domain D ∈ R2 . Then     3  ∂G ∂F F (u, v)du + G(u, v)dv = − dudv (5.1) ∂u ∂v C S

LEMMA 5.1.

where 1. C is a sectionally smooth boundary of the surface of integration S. 2. The positive direction of contour C is consistent with the definition of the positive direction of the vector = n

(∂r/∂u) × (∂r/∂v) ∂r/∂u × (∂r/∂v)

where r is the position vector. PROOF.

(See Section E.5.1 for proof.)

For the special case in which the surface of integration S lies in the (x, y)plane, Green’s lemma is stated in terms of x and y, that is, in (5.1) u and v are replaced by x and y, respectively. In this case, the positive direction of the contour is counterclockwise. Equivalently, the positive contour direction is chosen such that the region in S is always to the left of the contour’s path.

EXAMPLE 5.1.

Consider F (x, y, z) and G(x, y, z) given by F (x, y, z)

=

x2 + y

G(x, y, z)

=

2x + 3y − z + 5

Let the surface of integration to be the top portion of the unit sphere with 0 ≤ θ ≤ π/4 and 0 ≤ φ ≤ 2π, as shown in Figure 5.1. Using the parameterization based on the spherical coordinates, u = φ and v = θ, x = sin(v) cos(u)

;

y = sin(v) sin(u)

;

z = cos(v)

and F (u, v)

=

(sin(v) cos(u))2 + sin(v) sin(u)

G(u, v)

=

2 sin(v) cos(u) + 3 sin(v) sin(u) − cos(v) + 5

206

Vector Integral Theorems

Figure 5.1. The surface of integration given by the top of a unit sphere.

which then yield   ∂F = cos(v) sin(u) + 2 sin(v) cos2 (u) ∂v ∂G = sin(v) (3 cos(u) − 2 sin(u)) ∂u and the surface integral in the right-hand side of (5.1).   π/4  2π  ∂G ∂F π − dudv = − ∂u ∂v 2 0 0 The closed contour in the (u, v) plane is shown in Figure 5.2. The positive direction is counterclockwise, and it yields a normal vector of points in S that is outward from the center of the sphere. Based on Figure 5.2, the line integrals can be calculated to be  2π  0  3 π π F (u, v)du = F (u, 0) du + F u, du = − 4 2 C 0 2π 3

 G(u, v)dv

π/4

=

C

0

 =

 G (2π, v) dv +

0

π/4

G (0, v) dv

  √  √ 3 2 π 3 2 π − +2+5 + −2−5 =0 2 4 2 4

Combining all the results, we see that Green’s lemma applies, that is,  3   3 ∂G ∂F Fdu + Gdv = − dudv ∂u ∂v C C S

Figure 5.2. The closed path of integration in the (u, v) plane.

5.1 Green’s Lemma

207

Figure 5.3. The surface of integration with holes: (a) the original surface S and contour C, (b) S1 , obtained by removing S2 , (c) the continuous contour 4 C1 for 4 S1 , (d) due to the cancelation 4 of line integrals in oppositely directed segments, C1 = C − C2 .

Let 1 and 2 be two opposite paths, that is, they travel through the same set of points but having switched the end points. Then we note that     F (u, v)ds = − F (u, v)ds → F (u, v)ds + F (u, v)ds = 0 1

2

1

2

We can then use this fact to evaluate surface integrals in surfaces that contain holes. For illustration purposes, consider the surface shown in Figure 5.3b. A surface S1 was obtained by removing S2 from the original surface S. In Figure 5.3c, a continuous contour C1 can be generated as the boundary of S1 . At some point along the outer boundary, contour C1 cuts a path to reach the inner hole. The contour then traverses the boundary of this hole. After reaching the point where the contour first entered the hole, the contour C1 retraces the same cutting path backward to continue tracking the outer path.

208

Vector Integral Theorems

The two line integrals that have traversed the same path segments, but in opposite directions, will cancel each other out. Thus, as shown in Figure 5.3d, the line integral along C1 will be equal to the line integral along C minus the line integral along C2 , which is the contour of the removed surface S2 , that is, 3 3 3 f ds = f ds − f ds (5.2) C1

C

C2

where f is any appropriate function. From the definition of surface integrals, we know that    g dS = g dS − g dS (5.3) S1

S

S2

where g is any appropriate function. Then applying Green’s lemma and (5.2) to (5.3),    3 3 ∂G ∂F − dudv = (Fdu + Gdv) − (Fdu + Gdv) ∂u ∂v S1 C C2 3 = (Fdu + Gdv)

(5.4)

C1

Thus Green’s lemma can be applied to surfaces with holes by using the strategy of cutting through the region to connect the outer contour with the inner contours of the holes.

5.2 Divergence Theorem Consider a vector field f . The divergence theorem1 states that the volume integral of the divergence ∇ · f in a given volume region V can also be evaluated indirectly by taking the surface integral of the flux of f , that is, f · n, over the surface S that bounds the region V . In other words, the net sum of sinks and sources of a property inside a volume region can be evaluated by calculating the total flux of that property coming out of the surface of that region. THEOREM 5.1.

Let f (x, y, z) be a vector field which is differentiable in V then    dS = f·n ∇ · f dV S

PROOF.

(See Section E.5.2 for proof.)

The divergence theorem can be applied to (4.55) and (4.57) to yield,    dS = ψf · n (∇ψ · f + ψ∇ · f ) dV S

    dS f×g ·n S 1

(5.5)

V

V

=

 

   g · (∇ × f ) − f · ∇ × g dV

(5.6) (5.7)

V

The theorem is also known as the Gauss-Ostrogradski-Green divergence theorem. However, because the same names appear in several versions of the divergence theorem, we just call the main theorem “divergence theorem” and leave the names to their specific forms (e.g., Green’s theorem and Gauss’ theorem).

5.2 Divergence Theorem





 dS ψn

=

S



209

∇ψdV

(5.8)

∇ × f dV

(5.9)

V



 × f dS n

=

S

V

Equations (5.8) can be obtained from (5.6) by setting f to be a constant vector. Likewise, (5.9) can be obtained from (5.7) by setting g to be a constant vector. Note, however, that the integrands in (5.6) and (5.7) are scalar fields, whereas the integrands in (5.8) and (5.9) are vector fields. By letting f = ∇φ in (5.6), we obtain Green’s theorem, also known as Green’s identities. THEOREM 5.2.

Let φ and ψ be scalar functions that are twice differentiable. Then       dS = ψ∇ 2 φ dV (5.10) (ψ∇φ) · n (∇φ · ∇ψ) dV + S



V



 dS (ψ∇φ − φ∇ψ) · n

=

S

PROOF.

V

  ψ∇ 2 φ − φ∇ 2 ψ dV

(5.11)

V

(See Section E.5.3 for proof.)

One could also view the divergence theorem as a reduction in the dimension of the integration, that is, a volume integral on one side while a surface integral on the other. Thus if the volume integration is difficult to obtain, the reduction to a surface integral would usually make it easier to evaluate, at least numerically. However, there are several instances in which volume integration would actually be easier to obtain than the surface integral. This is specially true if the divergence operation simplifies several terms in the integrand, as shown in the next example.

Consider the vector field     = ax x2 + bx x + f (y, z) δx + ay y2 + by y + g (x, z) δy   + azz2 + bzz + h (x, y) δz

EXAMPLE 5.2.

f

where ax , ay , az, bx , by , and bz are constants. Let the surface of integration S  dS could be very difficult be a sphere of radius ρ. The surface integral S f · n to evaluate directly, especially when f , g, and h are complicated functions. However, the divergence of f yields     ∇ · f = 2 ax x + ay y + azz + bx + by + bz Using the divergence theorem,        dS = 2ax f·n xdV + 2ay ydV + 2az zdV + bx + by + bz dV

 S



=

V



V

4 bx + by + bz πρ3 3

V

V

210

Vector Integral Theorems

Another application of the divergence theorem, via the Green’s theorem given previously, is in the development of “weak solutions” needed by the finite elements methods. Briefly, the finite elements method partitions domains in which the first derivatives are satisfied. However, at the edges where the elements are patched together, the smoothness is no longer guaranteed. Green’s theorem allows for volume integrals of Laplacians such as ∇ 2 φ to be replaced by the surface integrals involving gradients. Details for the weak-solution formulation of the finite element methods are covered later in Section 14.1. A very important application of the divergence theorem is the Gauss’ theorem, which is also useful in solving partial differential equations. THEOREM 5.3.

 S

PROOF.

1  dS = δ ·n r2 r

5

0

if origin is outside of S

4π

if origin is inside of S

(5.12)

(See Section E.5.4 for proof.)

5.3 Stokes’ Theorem and Path Independence Another important theorem is the Stokes’ theorem. It gives a relationship between the curl of a vector field f in a region S and the tangential projection of f along the boundary of S. It can also be seen as a generalization of Green’s lemma for a plane.2 THEOREM 5.4. Let f be a differentiable vector field in a surface S bounded by the closed curve C, where S is a connected and sectionally smooth surface. Then 3   dS f · dr = (∇ × f ) · n (5.13) C

PROOF.

S

(See Section E.5.5 for proof.)

The line integral, 3

3

f · dr = C

f · t ds

C

is known as the circulation of f along the closed path C. Thus (5.13) simply states that the sum of all the curls of f fluxing out of the surface S bounded by C is equal to the circulation of f along C (see Figure 5.4). Stokes’ theorem has several application in physics, including one approach in the development of Maxwell’s equations for electric and magnetic intensity. Another application is to use it to assess path independence of line integrals.

2

In fact, based on exterior calculus, a general Stokes’ theorem can be developed that also generalizes the divergence theorem.

5.3 Stokes’ Theorem and Path Independence

211

Figure 5.4. A visual description of stokes theorem.

Definition 5.1. Let f be a given vector field and r be the position vector, then the line integral  f · dr IC = (5.14) C

is said to be path independent in a region V if, for any pair of curves, C1 and C2 inside the region V , IC1 = IC2

(5.15)

where C1 and C2 are continuous and sectionally smooth curves that have the same initial point (xi , yi , zi ) and the same end point (x f , y f , zf ). When V is not specified, then it is usually understood that the path independence refers to the whole 3D space. Consider two paths C1,AB and C2,AB in region V that do not intersect each other except at the start and end points. If the line integrals are independent of path,   f · dr = f · dr (5.16) C1,AB

C2,AB

which could be combined into one integral,   0 = f · dr −  = 3 0

C1,AB

C1,AB −C2,AB

=

f · dr

C2,AB

f · dr

f · dr

(5.17)

C

where C = C1,AB − C2,AB is a simple closed path, as shown in Figure 5.5. With any choice of C1,AB and C2,AB (so far assumed to be nonintersecting at midpath), by reversing the path direction of C2,AB, a closed path C is generated. Path independence guarantees that the line integral using the closed path will have to be zero. Alternatively, we could have rearranged (5.16) to be  3 f · dr = 0 = f · dr (5.18) C2,AB −C1. AB

C

where C = C2,AB − C1,AB is also a simple closed path but in the opposite direction of C. Now consider the situation shown in Figure 5.6 where path C1,AB and C2,AB might intersect each other somewhere between the end points, say, at point D. Path C1,AB can then be partitioned to be the sum of two subpaths: C1,AD and C1,DB. Likewise, path C2,AB can also be partitioned to be the sum of two subpaths: C2,AD and C2,DB.

212

Vector Integral Theorems

Figure 5.5. Two paths with the same start point A and end point B.

Equation (5.16) can now be expanded to be   f · dr − 





f · dr + C1,AD



f · dr − 

C1,DB

f · dr − C1,AD

C1,AB



f · dr − C2,AD



f · dr + C2,AD



C1,AD −C2,AD

 

f · dr + 3

=

0

f · dr

=

0

f · dr

=

0

f · dr

=

0

f · dr

=

0

C2,DB

f · dr − C1,DB

f · dr

C2,AB

C2,DB

C1,DB −C2,DB

3

f · dr − CAD

(5.19)

CDB

Figure 5.6. Two paths with the same start point A and end point B plus an intersection at point D.

5.3 Stokes’ Theorem and Path Independence

where CAD is the closed path formed by adding path C1,AD and reverse of path C2,AD. Similarly, CDB is the closed path formed by adding path C1,DB and reverse of path C2,DB. Because the definition of path independence applies to the subpaths, the two closed paths will each generate a zero line integral also. Thus the condition for path independence is equivalent to having the line integral of any closed paths in region V all be zero, including whichever direction the closed path takes. Having established this equivalence, we can use Stokes’ theorem to determine path independence. Recall that Stokes’ theorem relates the line integral in a closed path to a surface integral involving the curl of vector field f : 3   · (∇ × f ) dS f · dr = n (5.20) C

S

If ∇ × f = 0 for all points in region V , then Stokes’ theorem suggests that this condition will also guarantee path independence, because for any surface S inside region V , a zero curl implies a zero line integral on the left-hand side of (5.20). One more detail is still needed, however. Stokes’ theorem requires that the integrand in the surface integral be bounded. This means that in the chosen region V , no singularities of the integrand can be present. In relation to the closed path integral, this kind of region is formally referred to as a simply connected region. Definition 5.2. A region V is simply connected if any simple closed path in the region can be continuously deformed to a single point. Otherwise, the region is multiply connected. A set of examples is shown in Figure 5.7. The first case in Figure 5.7(a) is a full rectangular box. In this case, we see that any closed path contained in the region can be deformed continuously into a point inside the box. The second case is shown in Figure 5.7(b). For this case, a cylindrical subregion has been removed from the center. Even though there are some closed paths that could deform to a point, the existence of at least one closed path that will not continuously deform to a point is sufficient to have this region be multiply connected. The third case is given in Figure 5.7(c). In this case, the rectangular box region has a spherical subregion removed. However, unlike case (b), any closed path inside the region of case (c) can still be continuously deformed to a point. Thus the third case is a simply connected region. We can now summarize the preceding discussion in the following theorem: Let ∇ × f = 0 inside a simply connected region V ; then the integral f · d r is independent of path inside region V . C

THEOREM 5.5.



EXAMPLE 5.3.

Consider the following vector fields:

h

=

g

=

y δx + x δy + 2z δz   2 x + y2 + z2 δx + 2xy δy + (y − z) δz

f

=

−

y+2 x−1 δx + δ + 2 δz 2 2 (x − 1) + (y + 2) (x − 1)2 + (y + 2)2 y

213

214

Vector Integral Theorems

Figure 5.7. Examples of simply and multiply connected regions. (a) Solid rectangular region: simply connected; (b) solid rectangular region with a cylindrical subregion removed from the center: multiply connected; (c) solid rectangular region with a spherical subregion removed from the center: simply connected.

and the following paths with the parameter t ranging from 0 to 1:

Path C1:

Path C3:

x

=

3 + cos(3πt)

x

=

−12t2 + 10t + 4

y

=

3 + sin(3πt)

y

=

3 + 4t(1 − t)

z

=

2t

z

=

2 − 2(1 − t)2

x

=

10t2 − 11t + 1

Path C2:

x

=

−6t2 + 5t + 1

y

=

2t2 − 5t

y

=

2t2 − 5t

z

=

−6t2 + 7t

z

=

10t2 − 9t

Path C4:

Figure 5.8 shows paths C1 and C2 . Note that C1 has a helical form. Figure 5.9 shows paths C3 and C4 together. Included in the figure is a line described by (x, y, z) = (1, −2, z). This line includes the singular point of vector field f . Thus the line is only relevant when considering f .

5.4 Applications

Figure 5.8. Paths C1 and C2 .

 We cantabulate the results of calculating the various type of line integrals of C h · dr, C g · dr, and C f · dr for different paths. This is shown in Table 5.1, together with curl of the respective vector fields. As expected, the line integrals for h are path independent because the curl is zero and the whole 3D space is simply connected. For g, because the curl is not zero, the line integrals depend on the path. Finally, for f , because there exists a simply connected region that covers C1 and C2 , the two line integrals are expected to be equal. However, as we increase the region to contain paths C3 and C4 , the region (after removal of regions containing singularities) is multiply connected. Thus the theorem states that we expect that path independence is no longer guaranteed, even though the curls along paths C3 and C4 are zero.

5.4 Applications In this section, we discuss two major applications of the vector integral theorems. The first application mainly uses the divergence theorem for obtaining the various conservation laws. The second application is in the field of electrodynamics. Note that the main activity here is to obtain integral and differential equations that describe the physical laws. The solution of these differential equations will not be treated here. Instead, the analytical and numerical solutions of these equations are discussed in later chapters.

Figure 5.9. Paths C3 and C4 . Also included here is the line (1, −2, z).

215

216

Vector Integral Theorems Table 5.1. Line integrals based on different vector fields and paths Paths Start: (4, 3, 0) End: (2, 3, 2)

Start: (1, 0, 0) End: (0, −3, 1)

Vector fields

C1

C2

C3

C4

Curl

h g f

−2 −36.062 4.343

−2 −45.2 4.343

1 −4.4 −1.927

1 −11.333 4.356

0 −δx + 2xδy 0

5.4.1 Conservation Laws and Balance Equations One of the major uses of vector analysis and the integral theorem is to use the law of conservation and balance equations within a closed region based on volume integrals (weak forms) to obtain differential vector laws (strong equations). We start by considering the rate of change of a quantity φ(V ) in a fixed volume V enclosed by a closed surface S. Often these quantities φ can be represented instead by an φV associated density, which we denote by  φV . For instance, if φ is the mass, then  will be the density ρ. The general rate of change equations specified by the boundary of V is now ⎛

⎞ Rate of Change ⎝ ⎠ of φ inside V

⎛

=

⎞ ⎛ ⎞ Net flux of φ Internal rate of generation ⎠+⎝ ⎠ −⎝ out of V of φ across surface S inside V ⎛ ⎞ ⎜ External effects ⎟ +⎝ ⎠ on V and S

(5.21)

Equation (5.21) is quite general and applicable to several objects being considered. In integral form, we have ∂ ∂t

 V

 φV dV = −

 S

  dS − φV v · n





 dS + f·n S

 G dV +

V

 EV dV +

V

ES dS S

(5.22) where G is the generation of φ per unit volume, whereas EV and ES are external effects on φ per unit volume and per unit surface area, respectively. Here we have also separated the convective flux  φV v from other mechanisms of fluxes described by f . We treat some of the major applications, including mass, momentum, chemical component, and energy. For each of these cases, each term (e.g., flux terms and external effects) will vary and requires extra constitutive equations obtained from physical and empirical laws (often constrained under specific conditions). 1. Mass Balance Here, we have φ = m, which is mass m; then the density is  φV = ρ. The flux of mass out is given only by convective flow normal to the surface boundary,

5.4 Applications

that is,

217

  dS ρv · n

Net flow of mass out of S = S

 is the unit normal vector in S pointing outward from dS. Because mass where n cannot be created nor destroyed, we have G = 0, EV = ES = 0 in (5.22). Thus (5.22) becomes   ∂  dS ρdV = − ρv · n ∂t V S Because V is assumed fixed, we can move the time derivative inside the integral.3 Then, after applying the divergence theorem, we obtain    ∂ρ + ∇ · ρv dV = 0 (5.23) ∂t V which is the integral form of the continuity equation. Because V , has been set arbitrarily, we can extract the integrand and obtain ∂ρ + ∇ · ρv = 0 ∂t

(5.24)

which is the differential form of the continuity equation. Using the substantial derivative operator defined in (4.65), that is, D/Dt = ∂/∂t + v · ∇, (5.24) can be rewritten as Dρ + ρ∇ · v = 0 Dt 2. Momentum Balance Let φ = v be momentum per unit mass and  φV = ρv becomes momentum per unit volume. We now assume that there is no internal generation of momentum (e.g., no reactions or explosions). Furthermore, we can take the external forces to be due only to pressure normal to surface S and gravitational acceleration g acting on the volume V . Thus the external effects are given by    dS + External effects = − p n ρ g dV S

V

The momentum flux has two components: one is due to convective flow, and the other is due to the stress at the boundary. The flow of momentum out across S is given by   ) dS Convective momentum flow out = (ρv) (v · n S

The other flux, also known as the molecular momentum flux, is due to the material stress traction vector and is given by the stress vectors pointing outward of S. Based on the discussion given in Example 4.4, we have   dS Molecular momentum flow out = T · n S 3

If V is not fixed, one needs to apply Leibnitz rules, as discussed in Section 5.5.

218

Vector Integral Theorems

where T is the stress tensor field. Combining the various terms, (5.22) now becomes       ∂  dS − p n  dS + ρ v dV = − ρvv+T ·n ρ g dV ∂t V S S V After applying the divergence theorem, and consideration that V is fixed but arbitrary, we arrive at ∂ (ρv) = −∇ · (ρ v v) − ∇ · T − ∇ p + ρg ∂t

(5.25)

Equation (5.25) is known as the Cauchy equation of motion4 or, in terms of the substantial time derivative operator, D ∂ = +v·∇ Dt ∂t (5.25) together with (5.24) (see Exercise E4.18) will be reduced to Dv = −∇ · T − ∇ p + ρg (5.26) Dt Two special cases are often used in the study of fluid dynamics. The first is when the material is incompressible and Newtonian, and we have ∇ · T = −μ∇ 2 v (see exercise E4.19). This reduces (5.26) to ρ

Dv = −∇ p + μ∇ 2 v + ρg (5.27) Dt The other special case is when ∇ · T is negligible, for example, for an inviscid fluid, then (5.26) reduces to Euler’s equation of motion, ρ

ρ

Dv = −∇ p + ρg Dt

(5.28)

3. Energy Balance Let φ =  E be the energy per unit mass; then  φV = ρ E. The energy per unit mass is the sum of three terms: the specific internal energy  u, the specific kinetic 2 energy v /2, and the specific potential energy e p , that is, 2

v + ep 2 The flow of energy due to convection across surface S is given by   ) dS E (v · n Flow of energy through S = ρ  E = u+

S

According to the first law of thermodynamics, there is no internal generation of energy.5 For the external effects, there is the heat energy coming from the surroundings and the net work done by the surroundings. Thus we have    dS + External rate of heat input = − q · n  dV S 4

5

V

As noted during the derivation, the assumptions used (e.g., no internal generation, no other body forces such as magnetic effects are present, etc.) need to hold when using (5.25); otherwise extra terms are needed. In some texts, they consider heat generated by reaction as a generation term. In our case, we consider heats of reaction and other latent heats as included in the specification of internal energy.

5.4 Applications

219

where q is the heat flux due to heat transfer and  is the rate of heat input per volume due to other mechanisms such as radiation, electric field, and so forth. The net work done by the surroundings is given by     dS p v+T ·v ·n Rate of net work done by surroundings = − S

Then, after applying the divergence theorem and including the equation of continuity to put it in terms of substantial time derivative, (5.21) becomes   D E = −∇ · p v − ∇ · T · v − ∇ · q +  (5.29) Dt known as the total energy balance equation. For the special case of potential energy being based only on gravity, we have ∇ e p = −g, which can be assumed to be independent of time. Then the substantial time derivative of  e p becomes D ep = −ρv · g (5.30) Dt We could also find the substantial rate of change of kinetic energy by taking a dot product of the equation of motion given in (5.26) by v. Doing so, we can combine the substantial derivative of both kinetic and potential energy to be     D 1 2 ρ v + ep = −v · ∇ p − v · ∇ · T Dt 2       = − ∇ · p v − p (∇ · v) − ∇ · T · v − T :∇v (5.31) where we used the identity given in (4.100), with the last term being  ∂v j T :∇v = T ij ∂i i,j =x,y,z Equation (5.31) is also known as the mechanical energy balance equation. We can then remove two terms, namely the substantial time derivative of the specific kinetic and potential energy, from the total energy balance in (5.29) by using (5.31) to obtain   D u = −p (∇ · v) − T :∇v − ∇ · q +  ρ (5.32) Dt which is known as the thermal energy balance equation. As a simple application, consider the energy balance for a stationary solid in which the only mode of heat transfer from surroundings is via conduction. Applying the equation for internal energy in terms of constant heat capacity Cv with v = 0 ρ

D u ∂ Tˆ = ρCv Dt ∂t

where we use Tˆ for temperature. For the heat transfer, we use Fourier’s law, that is, q = −k∇ Tˆ

220

Vector Integral Theorems

where k is the thermal conductivity coefficient. With  = 0, we end up with the thermal diffusion equation given by ∂ Tˆ k  = α∇ 2 Tˆ = ∇ · ∇T ∂t ρCp

(5.33)

where α = k/(ρCp ) is the thermal diffusivity coefficient. 4. Component Balance We limit our discussion to binary component systems containing substance A and substance B. The balance will then be on φ being the moles of A, for which the density becomes  φV = cA, which is the concentration of A in units of moles per volume of solution. The flow out due to molar flux across surface S is now given by two terms, one by convection and the other by diffusion,   dS Net flow out of A = (cAv + D) · n S

where the velocity v is the molar average velocity defined by v =

cAvA + cBvB cA + cB

and D is the flux due to diffusion. The internal generation term will be the net rate of production of A due to reaction.6 This is given by  Rate of net generation of mole A via reaction = RAdV V

Substituting all the terms to (5.21) and then applying the divergence theorem, we obtain the binary component balance equation given by ∂cA + ∇ · (cAv ) = −∇ · D + RA ∂t

(5.34)

Using Fick’s law, D = −c DAB∇xA where c = cA + cB is the total concentration, xA is the mole fraction of A and DAB is the diffusivity of A in the AB solution. With the added assumption of constant density and constant diffusivity DAB, (5.34) will reduce to the reaction-diffusion equation, ∂cA + v∗ · ∇cA = DAB∇ 2 cA + RA ∂t

(5.35)

5.4.2 Electromagnetics One of the important applications of vector analysis, in fact a major impetus for the development of vector calculus, is the study of electricity and magnetism. The 6

Because we are treating only binary mixtures, we are assuming only the reaction A → B.

5.4 Applications

221

Table 5.2. Terms and relations in electromagnetics Terms

Notation

Formula

Fields Electric field

E

Magnetic field

H

Charge density (field)

Q Flux Densities

Electric flux density

D

Magnetic flux density

B

Conduction current density

Jc

σE

Displacement current density

Jd

∂D ∂t

Current density

Jc + Jd

J Integrals and Fluxes

  dS B·n

φ

Magnetic flux

 Current

S

 dS J·n

i 

Total charge

S

q

Q dV V

Parameters Permisivity



Permeability

μ

Conductivity

σ

different fields, flux densities, and fluxes7 are defined and given in Table 5.2, where J = Jc + Jd is Maxwell’s decomposition of the current density.8 Based on experimental studies, several laws were obtained that relate the different items in Table 5.2. These are  =

−

H · dr

=

i

(5.37)

 dS D·n

=

q

(5.38)

 dS B·n

=

0

(5.39)

C



∂φ ∂t

E · dr

(5.36)

C



S



S

7

8

As mentioned earlier in Example 4.3, the term “flux” in electromagnetics refers to a scalar quantity that integrates the flow of a flux density vector field through a given closed surface S. The term “flux” in fluid transport equations refers to a vector field. We temporarily suspend our convention of using small bold letters for vector fields to comply with the usual conventions of electromagnetics. Likewise, please note that i is set as current and not the imaginary number.

222

Vector Integral Theorems

 is the outward unit normal where r is the position vector along curve C and n vector to the surface S. Equations (5.36) and (5.37) are known as Faraday’s law and Ampere’s law, respectively, whereas (5.38) and (5.39) are known as Gauss’ law for electric fields and Gauss’ law for magnetic field, respectively. Note that Faraday’s law and Ampere’s law involve the line integrals of the electric field and magnetic field projected along closed paths C and yield the work resulting from each of these fields. On the other hand, both Gauss’ law for electric fields and magnetic fields are surface integral of a closed surface S, where the surface integral in (5.38) depends on the total charge inside the region bounded by S, whereas the surface integral in (5.39) is always zero. Let us assume a simple linear relationship between E and D and between B and H given by D

=

E

(5.40)

H

=

1 B μ

(5.41)

with  and μ constant. Applying Stokes’ theorem to Faraday’s law and Ampere’s law, and applying the divergence theorem to both of the Gauss’ laws, while assuming C, S, and V to be fixed, will yield the following:   ∂  dS E · dr = − B·n ∂t S C   ∂B  dS = −  dS ·n (5.42) (∇ × E) · n S S ∂t 



H · dr C



 dS J·n

= S

 dS (∇ × H) · n

=

   ∂E  dS σE +  ·n ∂t S

=

q

S

(5.43)

  dS D·n S





(∇ · D) dV

=

Q dV

V

(5.44)

V

  dS B·n

=

0

(∇ · B) dV

=

0

S



(5.45)

V

Because the value for C, S, and V are arbitrary, equations (5.42) to (5.45) reduce to the set of equations known as Maxwell’s equations for electricity and magnetism: ∇ ×E

=

∇ ×H

=

∂H ∂B = −μ ∂t ∂t ∂E σE +  ∂t

−

(5.46) (5.47)

5.4 Applications

223

∇ ·D

=

∇ · E = Q

(5.48)

∇ ·B

=

μ∇ · H = 0

(5.49)

where the rightmost equations of the first two equations show the coupled relationships between the fields E and H. These equations could be uncoupled by further taking the curl of (5.46) and (5.47), together with (5.48), (5.49), and identity (4.62), ∇ × (∇ × E) = ∇ (∇ · E) − ∇ 2 E

=

1 ∇Q − ∇ 2 E 

=

∇ × (∇ × H) = ∇ (∇ · H) − ∇ 2 H

=

−∇ 2 H

=

∂ (∇ × H) ∂t    2  ∂E ∂ E −μσ − μ ∂t ∂t2 −μ

  ∂(∇ × E) σ ∇ ×E + ∂t    2  ∂H ∂ H −μσ − μ ∂t ∂t2

which could be written in operator form as   1 M E = − ∇Q    M H = 0

(5.50) (5.51)

where M = μ

∂2 ∂ + μσ − ∇ 2 2 ∂t ∂t

EXAMPLE 5.4. For the special case of static electric fields, that is, E = E(t), the operator M reduces to the negative Laplacian, −∇ 2 , yielding

1 ∇Q  However, this consists of solving three differential equations because it is a vector differential equation. An interesting alternative is to observe that (5.46) under static conditions reduces to ∇ 2E =

∇ ×E=0 Based on Theorem 5.5, work done by E along a path inside a simply connected region will be path independent or, equivalently, E is a conservative field. For conservative fields, such as E in this case, we can make use of the vector differential identity (4.60), that is, ∇ × ∇ψ = 0, and let E = ∇ψ, where ψ is now the unknown potential field. Substituting this representation into (5.48), we have Q  which has a form known as Poisson’s equation. For the additional condition that Q = 0, this reduces to ∇ · E = ∇ 2ψ =

∇ 2ψ = 0

224

Vector Integral Theorems

a form known as Laplace’s equation. Once ψ has been solved from either the Poisson equation or Laplace equation together with the given boundary conditions, E can be obtaind by simply taking the gradient of ψ. Unfortunately, from (5.47), one observes that even for static magnetic fields, H is generally nonconservative. Nonetheless, one can still use M = −∇ 2 , under static conditions, and solve ∇ 2H = 0

5.5 Leibnitz Derivative Formula The last integral formula included in this chapter is the Leibnitz formula, which gives the method for evaluating the derivative of an integral with respect to a parameter α. THEOREM 5.6.

d dα

Given a function F (α, x) that is differentiable in x and α, then 



h(α)

F (α, x) dx

h(α)

=

g (α)

g (α)

∂F (α, x) dx ∂α

+F (α, h(α)) PROOF.

∂h(α) ∂g (α) − F (α, g(α)) ∂α ∂α

(5.52)

(See Section E.5.6, item 1, for proof.)

In three dimensions, we have Let f (x, y, z, α) be differentiable in an open region containing 3D region V (α). Then,    d ∂f ∂r · f (x, y, z, α) dV = dV + f (x, y, z, α) n dS (5.53) dα V (α) ∂α V (α) ∂α S(α)

THEOREM 5.7.

 is the unit outward normal vector, and where S(α) is the boundary surface of V (α), n r is the position vector of the points in the boundary S (α). PROOF.

(See Section E.5.6, item 2, for proof.)

In several applications of the Leibnitz rule, the parameter α is taken to be the time variable t. Thus it can be used to obtain integral balance equation around dynamic regions, that is, V = V (t) and S = S(t). Let us verify Liebnitz rule for the time derivative of the following volume integral  1  2  e−t    f (x, y, z, t) dV = t x2 + y2 + z2 dx dy dz

EXAMPLE 5.5.

V

0

0

0

5.6 Exercises

225

The derivative of this integral can be found by direct computation to be    d 2 10 f (x, y, z, t) dV = − 2t e−3t + (5.54) (1 − t) e−t dt V 3 3 which forms the left-hand side of (5.53). For the first term, that is, the volume integral on the right-hand side of (5.53), we have   1  2  e−t  2  ∂f 2 10 dV = x + y2 + z2 dx dy dz = e−3t + e−t (5.55) 3 3 V ∂t 0 0 0 As for the surface integral term on the right-hand side of (5.53), of the six possible faces, only one of the surface integrals is nonzero, which is the face at x = e−t whose normal is δx . Thus we have   1 2     ∂r 10 · f n dS = t e−2t + y2 + z2 −e−t dy dz = −2te−3t − te−t ∂t 3 S 0 0 (5.56) Adding the results of (5.55) and (5.56) will yield the same result given by (5.54).

5.6 EXERCISES

E5.1. Let a particle move along a path C due to an attractive force f directed toward the origin, f = kr. The tangential unit vector along the path is given by dr dr

δt =

1. With the projection of a force f = kr onto δt given by   dr f t = k r · dr 2 dr where k is a constant coefficient, show that the component of f that is normal to the path will be given by fn = k

dr

2

r − (r · dr) dr dr

2

2. With friction proportional to the normal component, friction = μ f n δt , show that the sum of the work due to f and the frictional force along path C is given by " ⎛ ⎞ 2 2  r dr − (r · dr)2 r · dr ⎝ ⎠ ds W =k +μ dr dr C 3. Evaluate the work due to f and its associated friction force for the cyclone path shown in Figure 5.10 given by √ t3 t3 3 3 y = sin (16πt) x = cos (16πt) z= t 4 4 4 from t = 0.1 to t = 1.5 in terms of μ and k.

226

Vector Integral Theorems

0.8

0.6

z

0.4

0.2

Figure 5.10. A cyclonic path.

0 0.5

0.5

y

0

0 −0.5

x

−0.5

E5.2. Let ψ (x, y, z) be the function ψ (x, y, z)

=

axx x2 + ayy y2 + azzz2 + axy xy + ayzyz + axzxz +ax x + ay y + azz + c

where axx , ayy , . . . axz, ax , ay , az, c are constants. Calculate the volume integral V ψ (x, y, z) dV , where the volume of integration is the sphere centered at the origin with a radius ρ. Determine the volume integral if the sphere V had its center at (x0 , y0 , z0 ) instead. E5.3. Consider the top in exercise E4.10 whose surface was described by x

=

sin (θ) cos (φ)

y

=

sin (θ) sin (φ)

z

=

1.5 cos (θ/2)

for 0 ≤ θ ≤ π and 0 ≤ φ ≤ 2π. 1. Find the volume of the solid. 2. Assuming a density function

  ρ(x, y, z) = 2 x2 + y2 + z2

Find the center of mass, c, given by  ρ r dV c = V V ρ dV where r is the position vector. E5.4. The moment of inertia of a rigid body with respect to an axis of rotation is given by  I = ρ D2 dV where ρ is the density field of the rigid body, and D is the distance of a point of the body from the axis of rotation (see Exercise E4.3 for the formula of distance of a point from a line). Find the moment of inertia of a sphere of radius R with the center at the origin, having a density field ρ(x, y, z) = 2x2 + y2 around an axis that passes through the points A = (1, 1, 0) and B = (1, 0, 1). E5.5. Prove the identities given in (5.6) through (5.9).

5.6 Exercises

227

E5.6. A vertical cylinder A of radius RA is parameterized by θ and s,where 0 ≤  θ ≤ 2π and −∞ < s < ∞ via (x, y, z) = RA cos(θ), RA sin(θ), s . Another horizontal cylinder B is parameterized by φ and t, where 0 ≤ φ ≤ 2π and  −∞ < t < ∞, via (x, y, z) = t, RB sin(φ), RB cos(φ) . Assume RB < RA. 1. Cylinder B cuts two separate pieces out of cylinder A. Show that one of the cut pieces   is a bounded surface given by (x, y, z) = RA cos(θ), RA sin(θ), s with new bounds −θ∗ ≤ θ ≤ θ∗ where     −1 RB    θ = sin R  ∗

A

and |s| ≤ s∗ (θ) where

6



∗

s (θ) = RB 1 −

RA RB

2 sin2 (θ)

This means that the surface integral for this cut piece is given by   θ∗  s∗ (θ) f (x, y, z)dS = g(s, θ) dsdθ −θ∗

S

−s∗ (θ)

2. Let functions F and G be given by G = x2 + 2y

F = 4z and

Verify Green’s lemma (5.1) with u = θ and v = s for the surface of the cut piece described Previously and RA = 1.5 and RB = 1. (Note: For the contour integral, the closed path can be parameterized by λ for points (θ, s) of the path, ranging from 0 to 1 with the following correspondences: (λ = 0) → (−θ∗ , 0), (λ = 0.5) → (θ∗ , 0), and (λ = 1) → (−θ∗ , 0)). E5.7. Let S be a closed surface, and show that the divergence theorem leads to the following identities:  1 V = n · r dS 3 S    n · ∇ × v dS 0 =  ∇ψ dV  V

S



S

=

V

∇ × v dV



nψ dS

=

n × v dS S

where V is the volume enclosed by S and v is a vector field. (Hint: For the last two identities, apply the divergence theorem on ψa and v × a, respectively, where a is a constant vector.) Note that the last two identities plus the divergence theorem can be combined into one mnemonic equation,       ∇  F dV = n  F dS (5.57) V

S

228

Vector Integral Theorems

where

Divergence Gradient Curl



[F]

dot product scalar product cross product

v ψ v

E5.8. Let C be a closed curve, and show that the following line integrals are zero: 3 L1 = a · dr C

3

r · dr

=

L2

C

3 =

L3

∇ψ · dr C

3 =

L4

(v∇w + w∇v) · dr C

where a is any constant vector, r is the position vector, and ψ, v, and w are scalar functions. E5.9. Let C be a closed curve, and using Stokes’ theorem, show that 3  ψ dr = n × ∇ψ dS C

S

(Hint: Apply Stokes’ theorem on ψa, where a is a constant vector.) Note that if we also rewrite Stokes’ theorem, using (4.52), as 3  dr · v = (n × ∇) · v dS C

S

then the two equations can be combined into one mnemonic equation, 3  dr  [F] = (n × ∇)  [F] dS (5.58) C

S

where

Cross − Divergence Cross − Gradient



[F]

dot product scalar product

v ψ

E5.10. Rewriting Euler’s equation of motion (5.28) as ∂v 1 + (v · ∇) v = g − ∇ p ∂t ρ For the case where ρ is constant, show that   ∂v p 1 2 ep + + − v × (∇ × v) = −∇  v ∂t ρ 2

5.6 Exercises

229

where g = −∇ e p , with  e p being the potential energy per unit mass. Then for steady state and irrotational velocity field, this equation reduces to the Bernoulli equation of motion,  ep +

p 1 + v ρ 2

2

= constant

along a streamline. E5.11. One can show that the stress tensor, in the absence of couple-moments, is symmetric. 1. Let the stress tensor field of a body be given by T and a body force per unit volume given by f . Show that for equilibrium, a necessary condition is given by ∇ · TT + f = 0

(5.59)

(Hint: The summation of forces along each coordinate should be zero. The forces include the surface integral of the force due to the stress tensor on the body surface and the volume integral of the body forces.) 2. Assuming no couple-moments, the total moments due to both body force and stress tensor on the surface should also be zero, that is,     r × T · n  dS + r × f dV = 0 (5.60) S

V

Show that after implementation of the divergence formula and (5.59), the stress tensors indeed need to be symmetric. E5.12. Obtain the differential volume at a point r under the paraboloid coordinate system described in Example 4.17. E5.13. Another orthogonal coordinate system is the torroidal coordinate system, (τ, σ, φ), with τ ≥ 0, −π ≤ σ ≤ π, and 0 ≤ φ ≤ 2π, described by the following equations: x=

a sinh(τ) cos(φ) cosh(τ) − cos(σ)

;

y=

a sinh(τ) sin(φ) cosh(τ) − cos(σ)

;

z=

a sin(σ) cosh(τ) − cos(σ) (5.61)

1. Obtain the differential volume dV in this coordinate system. 2. A torus-surface can be described by fixing τ to be constant, with τ → ∞ yielding a circle (torus with no volume) of radius a. Let a torus be defined by parameters R and C (with C > R), where R is the radius of a circle located in any plane containing the z-axis and whose center is located in the (x, y)-plane at a distance C away from the z-axis. Show that the value of a and τ that correspond to parameters R and C are given by   cosh(τ) − 1 −1 C and a = (R + C) τ = sinh R sinh(τ) 3. We want to determine the amount of property q flowing out of the surface of a torus in a flow field per unit area given by  2 −r v = 4 exp (5.62) δr 10

230

Vector Integral Theorems

where r is the spherical radius coordinate. Instead of having to find the unit normal vectors at the surface of the torus followed by the evaluation of surface integral, an alternative approach is to use the divergence and replace the problem with evaluation of a volume integral    ∇ · v dV (5.63) V

Find the total flow of property q out of a torus with R = 1 and C = 2 for the flow field per unit area given in (5.62). E5.14. A spherical solid of radius r containing substance s is decaying in time. The distribution of s is described by a time-dependent field   z 3 −t(x2 +y2 )/3 φ(x, y, z, t) = 0.8 e + 2r 2 where the center of the sphere is located at the origin. Suppose the radius of the sphere is also shrinking, symmetrically with respect to the center, due to erosion. The radius was found to be shrinking at a rate dr = −0.1r dt

r(0) = R = 10

1. Find r = r(t). 2. Determine the rate of change of s present in the solid at time t = 10. 3. Verify Leibnitz rule (5.53) for this rate of change for any time t > 0. E5.15. Consider the regions R1 through R3 and singularities contained in S1 through S4 given in Table 5.3. Determine which pairs (i.e., R1 with S1 removed, R1 with S2 removed, etc.) are simply connected. Table 5.3. Regions R and singularities S for E5.15 Regions

Singularities

R1:

−4 ≤ x ≤ 4 y≥0 −∞ < z < ∞

S1:

R2:

spherical r ≥ 2

S2:

R3:

cylindical r ≥ 2

S3: S4:

(x, y, z) = (0, 0, 0) Surface 2x + y − z = 0 Sphere with radius r = 0.1 and center at (4, 0, 0) Line passing through (1, 1, 2) and (2, 1, −1)

E5.16. Let f and g be given by f

=

g

=

h

=

2 δ (under spherical coordinates) r−1 r 2   δ (under cylindrical coordinates) sin θ + π3 r xzδx +

y δ + zδz y+3 y

5.6 Exercises

Determine whether the following integrals are path independent or not:  1. C f · dr , with path C restricted to the region: r ≤ 2 (where r is the spherical radius).  2. C g · dr , with path C restricted to the region: 0 ≤ θ ≤ π (where θ is cylindrical angle).  3. C h · dr , with path C restricted to the region: y ≥ 0

231

PART III

ORDINARY DIFFERENTIAL EQUATIONS

Several models of physical systems come in the form of differential equations. The main advantage of these models lies in their flexibility through the specifications of initial and/or boundary conditions or forcing functions. Although several physical models result in partial differential equations, there are also several important cases in which the models can be reduced to ordinary differential equations. One major class involves dynamic models (i.e., time-varying systems) in which the only independent variable is the time variable, known as initial value problems. Another case is when only one of the spatial dimensions is the only independent variable. For this case, it is possible that boundary conditions are specified at different points, resulting in multiple-point boundary value problems. There are four chapters included in this part of the book to handle the analytical solutions, numerical solutions, qualitative analysis, and series solutions of ordinary differential equations. Chapter 6 discusses the analytic approaches to solving first- and second-order differential equations, including similarity transformation methods. For higher order linear differential equations, we apply matrix methods to obtain the solutions in terms of matrix exponentials and matrizants. The chapter also includes the use of Laplace transforms for solving the high-order linear ordinary differential equations. Numerical methods for solving ordinary differential equations are discussed in detail in Chapter 7, including implicit and explicit Runge-Kutta methods and multistep methods such as Adams-Bashforth and Adams-Moulton and the backward difference formulas (BDF) methods. We also discuss some simple error-control approaches by adaptive time-interval methods. The second part of the chapter is devoted to the solution of boundary value problems such as the shooting method for both linear and nonlinear ordinary differential equations and the Ricatti method. The last part of Chapter 7 is a relatively brief, but crucial, discussion of difference equations and stability. The analysis of difference equations is important for the application of the Dahlquist tests to determine stability regions. Using the ideas of stability, one can better descirbe “stiff” differential equations. Chapter 8 discusses the qualitative analysis of differential equations such as phase-plane analysis; stability analysis of equilibrium points, including Lyapunov methods and linearization techniques; and limit cycles. This chapter also discusses bifurcation analysis based on one- or two-parameter systems. 233

234

Ordinary Differential Equations

Finally, in Chapter 9, we discuss the series solution methods of Frobenius for second-order and higher order linear differential equations containing variable coefficients. Among the most important applications of the series method are the solutions of the Legendre equations, the associated Legendre equations, and Bessel equations. These differential equations often result from the analytical solution of partial differential equations, which are covered in Part IV of the book.

6

Analytical Solutions of Ordinary Differential Equations

In this chapter, we discuss the major approaches to obtain analytical solutions of ordinary differential equations. We begin with the solutions of first-order differential equations. Several first-order differential equations can be transformed into two major solution approaches: the separation of variables approach and the exact differential approach. We start with a brief review of both approaches, and then we follow them with two sections on how to reduce other problems to either of these methods. First, we discuss the use of similarity transformations to reduce differential equations to become separable. We show that these transformations cover other well-known approaches, such as homogeneous-type differential equations and isobaric differential equations, as special cases. The next section continues with the search for integrating factors that would transform a given differential equation to become exact. Important special cases of this approach include first-order linear differential equations and the Bernoulli equations (after some additional variable transformation). Next, we discuss the solution of second-order differential equations. We opted to focus first on the nonlinear types, leaving the solution of linear second-order differential equations to be included in the later sections that handle high-order linear differential equations. The approaches we consider are those that would reduce the order of the differential equations, with the expectation that once they are firstorder equations, techniques of the previous sections can be used to continue the solution process. Specifically, we use a change of variables to handle the cases in which either the independent variable or dependent variable are explicitly absent in the differential equation. In addition, we also note that if the differential equation admits similarity transformations, a new pair of independent variable and dependent variable can be used to reduce the order. We also include the case of Euler-Cauchy differential equations in which we can transform the equation to one that is a linear differential equation with constant coefficients. Before going to higher order differential equations, we include a section that discusses some important topics when handling first- or second-differential equations. One topic is the general Ricatti equation, in which we actually increase the order of the differential equation to attain linearity. Another topic is the use of Legendre transformations to handle equations in which the derivatives are present in nonlinear forms, whereas the normal variables are in linear forms. Finally, we also discuss the issue of singular solutions, which appear only in nonlinear equations. Singular 235

236

Analytical Solutions of Ordinary Differential Equations

solutions do not contain any arbitrary constants, and they provide an envelop that helps determine regions of the domains where solutions exist. Next, we begin our discussion of higher order differential equations with the state-space formulation. Even when one decides to use numerical approaches, the initial step, more often than not, is to first recast the differential equations in the statespace forms. We limit our discussion of analytical solutions only to linear high order differential equations. The nonlinear cases are often handled much better using numerical methods. Nonetheless, even by limiting our problems to linear differential equations, we still have two cases: those with constant coefficient and those with nonconstant coefficients. For the linear equations with constant coefficients represented by a constant matrix A, the solutions are pretty standard, with the matrix exponentials eAt as the key element of the general solutions. This means we need the results of earlier chapters (i.e., Chapters 1 through 3) to help in evaluating the required matrices and functions. For instance, when the matrix A is diagonalizable (cf. Section 3.5), then eAt = Vet V −1 , where  is a diagonal matrix of eigenvalues and V is the matrix of eigenvectors. For the case in which A is not diagonalizable, we include a brief section discussing the application of the Cayley-Hamilton theorem to provide a finite-sum approach to evaluating the solution. For the case in which the system of linear high-order differential equations have nonconstant coefficients (i.e., A = A(t)), we have the generalization of the solution in the form of fundamental matrices, also known as matrizants. Although the evaluation of matrizants is difficult to find in general, some special cases do exist in which the matrizants can be found directly. For instance, when A(t)A(τ) is commutative, the matrizant will involve a simple matrix integral. More importantly, we use the concept of matrizants in the next chapter, when we solve boundary value problems of linear differential equations with nonconstant coefficients using numerical approaches. Next, we discuss the idea of decoupling of differential equations. This is possible when the matrix A is diagonalizable. The idea that the original system can be transformed into a set of decoupled differential equations is introduced in this section. The main issue is not so much the solution of the differential equations, because they are simple restatements of the solution involving exponential matrices eAt . The distinct feature of the decoupled system lies in its offering of an alternative space where the decoupling condition allows the tracking of solution in a one-dimensional space. Potentially, it allows for easier design, interpretation, and analysis of physical experiments. We include one such example in the form of the Wei-Prater method for the determination of kinetic constants of multiple interacting equilibrium reactions. Finally, we include a brief discussion of Laplace transform methods for the solution of multiple equations. Specifically, we show that this approach also yields the same results of the approach given in the earlier sections.

6.1 First-Order Ordinary Differential Equations In this section, we limit our discussion to the solution of first-order ordinary differential equations (ODEs) that can be cast either in the derivative form, given by dy = f (x, y) dx

(6.1)

6.1 First-Order Ordinary Differential Equations

237

or in the differential form, also known as the Pfaffian form, given by1 M (x, y) dx + N (x, y) dy = 0

(6.2)

The equivalence between the two forms can be obtained by setting f (x, y) = −

M (x, y) N (x, y)

(6.3)

Although both forms are useful, (6.1) has the additional interpretation (or constraint) of fixing x and y as the independent and dependent variable, respectively, due to the definition of derivatives. Conversely, (6.2) treats both variables as independent variables of an implicit solution, i.e., S(x, y) = C, where C is an arbitrary constant to fit initial or boundary conditions.2 When required, and if it is possible, an explicit solution can be attempted by rearranging S(x, y) = C to obtain a form y = y(x, C). For this reason, we predominantly treat the solution of (6.2), except where the given ODE can be identified more easily with standard (canonical) forms that are given in the derivative forms, such as the linear first-order ODE and Bernoulli equations. Most approaches fall under two major categories: those that are reducible to separation of variables approach and those that are reducible to exact differential approach. In this perspective, several techniques focus simply on additional transformations, for example, additional terms, multiplicative factors, or the change of variables, to reduce the problems into one of these categories. Suppose that after a set transformation is applied on x and y, new variables x and y are obtained resulting in a separable form given by  (  ( M x) d x+N y) d y=0

(6.4)

then the solution approach is known as the method of separation of variables, yielding     (  ( C (6.5) y) d y= x) d x+ N S ( x, y) = M with  C as arbitrary constant. However, if after the transformation to new variables, we obtain   ∂M ∂N = ∂ y ∂ x

(6.6)

known as the exactness condition, then we say that the transformed differential equation is an exact differential equation and the solution is given by     (  ( M x, y) d x + g ( y) = N x, y) d y + h ( x) S=  y held constant  x held constant (6.7) where g ( y) and h ( x) are determined by matching terms in the rightmost equation in (6.7). 1

2

A more general form is given by

  dy F x, y, =0 dx

which we discuss only in the context of Legendre transformations or singular solutions. There are solutions that do not allow for the inclusion of an arbitrary constant. These are known as singular solutions, which are discussed later in Section F.2.

238

Analytical Solutions of Ordinary Differential Equations EXAMPLE 6.1.

Consider the differential equation (x + y) dx + x dy = 0

(6.8)

∂M ∂N = 1 and = 1, which sat∂y ∂x isfies the exactness condition given by (6.8),and no transformations are needed. Thus the solution is given by With M(x, y) = x + y and N(x, y) = x, we have

x2 S(x, y) = + xy = C 2

→

  1 x2 y= C− x 2

Alternatively, if we let  y = y/x and  x = x, then (6.8) can be transformed to be 1 1 d x+ d y=0  x 2 y+1 which is a separable form. The solution then becomes     2  1 1 1 C x  y= − C → y= − 2  x2 x 2 2 which is the same solution with  C = 2C.

Remarks: 1. In the next section, we group several techniques as similarity transformation methods. Most of these techniques should be familiar to most readers. It turns out that they are just special cases of similarity transformations. 2. Later, in the section after next, we focus on the search for an integrating factor needed to achieve exactness. Unfortunately, it is often that a systematic search may yield equations that might even be more formidable than the original problem. Thus we simply outline the guiding equations from which some heuristics will be needed to make the approach more practical. 3. It is possible that the resulting integrals are still not easily reduced to closed forms. Thus numerical integrations may still be needed to obtain the solutions. 4. Although an integrating factor is sometimes hard to find, the use of exact differentials have resulted in other benefits. For instance, in thermodynamics, Caratheodory successfully used the reciprocal of absolute temperature 1/T as an integrating factor to show that the change in entropy, s, defined  as the integral of the ratio of differential heat transfer to temperature (i.e., δQ/T ), becomes a path-independent function.

6.2 Separable Forms via Similarity Transformations We begin with a general definition of symmetry transformations, under which similarity transformations are special cases.

6.2 Separable Forms via Similarity Transformations

239

Definition 6.1. A pair of transformations  x = x(x, y)

;

 y = y(x, y)

(6.9)

is called a symmetry transformation (pair) for a first-order differential equation M (x, y) dx + N (x, y) dy = 0 if, after substituting the new variable defined by (6.9), the new differential equation is given by M ( x, y) d x + N ( x, y) d y=0 that is, the same functionalities for M and N remain, except that x and y are replaced by  x and  y, respectively. Furthermore, we say that the new differential equation has attained symmetry based on the transformations. There exist general approaches to obtain symmetry transformations, and the notion of symmetry transformations can also be generalized for higher order differential equations. However, even for the first-order systems, the symmetry transformations are generally difficult to obtain, and in some cases they require the solution of partial differential equations. We limit our discussion only to a special class of symmetry transformations known as similarity transformations, which are among the simplest symmetry transformations to try. Definition 6.2. A pair of transformations & x = λα x

;

& y = λβ y

(6.10)

is called a similarity transformation (pair) for a first-order differential equation M (x, y) dx + N (x, y) dy = 0 if, after substituting (6.10), the new differential equation attains symmetry, given by M (& x,& y) d& x + N (& x,& y) d& y=0 where λ is called the similarity transformation parameter, and α and β are nonzero real constants. If a similarity transformation exists, that is, if one can find real values for α and β in (6.10) to attain symmetry, then we can combine the variables to obtain a new variable u known as the similarity variable or invariant given by u=

EXAMPLE 6.2.

& yα yα = & xβ xβ

Consider the differential equation given by 2  y + 2x2 dx − x3 dy = 0

Substituting x = λ−α& x and y = λ−β& y, we get  2 λ−2β−α & y + 2λβ−2α& x2 d& x − λ−3α−β& x3 d& y=0

(6.11)

240

Analytical Solutions of Ordinary Differential Equations

To attain symmetry, that is, to remove the presence of λ, we need β = 2α. In particular, we could set β = 2 and α = 1. Doing so, we have the invariant3 u=

yα y = 2 β x x

Having determined the invariant u, the next theorem guarantees that the original differential equation can be made separable. THEOREM 6.1.

Let the differential equation M (x, y) dx + N (x, y) dy = 0

y = λβ , where α = 0 and admit a set of similarity transformations given by& x = λα and& β = 0. Then, using the invariant u = yα x−β as a new variable, while maintaining x as the other variable, will transform the differential equation into a separable variables form given by 1 1 dx + du = 0 (α−1)/α x βu − αu G(u) where

PROOF.

⎧ M (x, ux) ⎪ ⎪ − ⎪ ⎪ N (x, ux) ⎪ ⎪ ⎪ ⎨   G(u) = 1/α  ⎪ ⎪ M x, uxβ ⎪ ⎪ ⎪  x(α−β)/α −  ⎪ ⎪ 1/α ⎩ β N x, (ux )

(6.12)

if α = β

if α = β

(See Section F.4.1 for proof.)

In Example 6.2, we found u = y/x2 to be an invariant for the differential equation 2  y + 2x2 dx − x3 dy = 0

EXAMPLE 6.3.

Applying Theorem 6.1, we have dx du − 2 =0 x u + 2u + 4 which is now separable, and the solution can be obtained as √  √ √ √   3 3 −1 tan (u + 1) = ln(Cx) → y = x2 3 tan 3 ln(Cx) − 1 3 3 3

If we had set β = 1 and α = 1/2, another possible invariant is given by √ y √ v= = u x In fact, any function of the invariant, θ (u), is also a valid invariant for the differential equation.

6.2 Separable Forms via Similarity Transformations

241

A large class of differential equations that immediately admits similarity transformations are the isobaric first-order differential equations given by the form y xn−1 f dx − dy = 0 (6.13) xn where f (η) is any differentiable function of η. This admits a similarity transformation with β = nα. In particular, with α = 1, β = n, and u = yx−n , Theorem 6.1 reduces (6.13) to the following separable form: dx du + =0 x nu − f (u)

(6.14)

Using the equation in Example 6.2 one more time, this differential equation can be rearranged to be y 2 x 2 + 2 dx − dy = 0 x

EXAMPLE 6.4.

which is an isobaric equation with n = 2 and f (η) = (η + 2)2 , where η = yx−2 . Based on (6.14) and u = yx−2 , we get dx du + =0 x 2u − (u + 2)2 which is the same separable equation obtained in Example 6.3. One special case of isobaric equations occurs when n = 1. These equations are known as the homogeneous-type first-order differential equations, that is, y dx − dy = 0 (6.15) f x which can be put into separable form in terms of the variables u = y/x and x, dx du + =0 x u − f (u)

(6.16)

Another set of equations for which similarity transformations may apply after some additional transformations is given by the form f (η)dx − dy = 0

(6.17)

where η is a ratio of affine terms, that is, η=

a1 x + a2 y + a3 b1 x + b2 y + b3

(6.18)

where a1 , a2 , a3 , b1 , b2 , and b3 are constants, and a1 b2 = a2 b1 .4 Let z = a1 x + a2 y + a3 and w = b1 x + b2 y + b3 . Then with η = z/w and      1 dx b2 −a2 dz = −b1 a1 dy dw a1 b2 − a2 b1 4

If a1 b2 = a2 b1 , one can just set η = a1 x + a2 y or η = b1 x + b2 y, whichever is nonconstant. This will immediately reduce the original equation to be separable.

242

Analytical Solutions of Ordinary Differential Equations

the original differential equation can be transformed into a homogeneous-type differential equation given by  b2 f

z w

 + b1

  z dz − a2 f + a1 dw = 0 w

which can be made separable under the variables w and ζ = z/w, that is, dw b2 f (ζ) + b1 = dζ w (a2 − b2 ζ) f (ζ) + (a1 − b1 ζ)

EXAMPLE 6.5.

(6.19)

Consider the differential equation   dy 5y + 1 2 = +1 dx x+2

Then applying (6.19) with w = x + 2 and ζ = (5y + 1)/(x + 2) = ζ, yields dw dζ = w 5 (ζ2 + 1) − ζ which is a separable differential equation whose solution is given by ' ( 2 1 ln(w) + ln(C) = √ arctan √ (10ζ − 1) 3 11 3 11 or in terms of x and y,  √  ⎡ √ ⎤ 3 11 tan 23 11 ln (C [x + 2]) + 1 ⎦ (x + 2) − 1 y=⎣ 50 5 where C is an arbitrary constant.

6.3 Exact Differential Equations via Integrating Factors We now consider the method of the inclusion of a function φ(x, y) known as the integrating factor to yield an exact solution. Recall that for M(x, y)dx + N(x, y)dy = 0 ∂M ∂N = . If the differential equation is not ∂y ∂x exact, we need to find an integrating factor, φ(x, y), such that the exactness condition is given by

φ(x, y)M(x, y)dx + φ(x, y)N(x, y)dy = 0 becomes exact, that is, ∂(φM) ∂(φN) = ∂y ∂x

→

  ∂N ∂M ∂φ ∂φ M −N =φ − ∂y ∂x ∂x ∂y

(6.20)

6.3 Exact Differential Equations via Integrating Factors

243

In general, the partial differential equation in (6.20) may be difficult to solve. A limited approach is to assume φ = φ(η) where η = η(x, y). This reduces (6.20) to    ⎞ ⎛  ∂M ∂N − ⎟ dφ ⎜ ∂y ∂x    ⎟ =⎜ dη (6.21) ⎝ ∂η ∂η ⎠ φ N − M ∂x ∂y If η could be found such that 

   ∂M ∂N − ∂y ∂x     = F (η) ∂η ∂η N − M ∂x ∂y then the integrating factor φ is immediately given by   φ = exp F (η)dη

EXAMPLE 6.6.

(6.22)

(6.23)

  Given xy − y2 dx + dy = 0, we have F =

∂M/∂y − ∂N/∂x x − 2y = (N)(∂η/∂x) − (M)(∂η/∂y) ηx − (xy − y2 ) ηy

With ηy = 1/y, the denominator can be made closer in form to the numerator. Thus η = ln(y) + f (x). Substituting η back to F ,  x  −2 − + y 2 F = df −x +y dx which can be made constant by setting can obtain the integrating factor: x2 η = ln(y) + 4

→

df x x2 − x = − , or f (x) = . Finally, we dx 2 4   1 x2 φ(x, y) = 2 exp − y 2

The conditions for exactness can be verified directly,  2 ∂φM x x ∂φN = − 2 exp − = ∂y y 2 ∂x and the solution to the differential equation is  2 7   1 x π x exp − + erf √ =C y 2 2 2 where C is the arbitrary constant of integration, and erf(z) is the error function defined by  2 z −t2 erf(z) = e dt π 0

244

Analytical Solutions of Ordinary Differential Equations

Based on (6.22), two special cases are worth noting. These are:     ∂M/∂y − ∂N/∂x Case 1 = p (x) → η = x → φ = exp p (x)dx N      ∂M/∂y − ∂N/∂x Case 2 = q(y) → η = y → φ = exp − q(y)dy M

EXAMPLE 6.7.

(6.24) (6.25)

For the differential equation given by   xy dx − x2 + y dy = 0

we have ∂M/∂y − ∂N/∂x 3 = M y Using (6.25), we get φ = y−3 . The solution is then given by x2 + 2y +C=0 2y2

The first-order linear equation is given by the standard form dy + P(x)y = Q(x) dx

(6.26)

This can be rearranged to be   P(x)y − Q(x) dx + dy = 0 which can be shown to fall under the case satisfying (6.24). Thus the integrating factor is   φ = exp P(x)dx (6.27) and the solution is given by     '  ( F (x, y) = y exp exp P(x)dx − P(x)dx Q(x) dx = C or

 (      '  y = exp − P(x)dx exp P(x)dx Q(x) dx + C

(6.28)

The component balance for a liquid reactant concentration CA in a continuously stirred reactor undergoing a first-order reaction A → P is described by  dCA F (t)  = CA,in (t) − CA − kCA dt V EXAMPLE 6.8.

6.4 Second-Order Ordinary Differential Equations

245

where F , V , k, and CA,in is the volumetric flow rate, reactor volume, specific kinetic constant, and inlet concentration of A, respectively. This can be rewritten in the standard form of a first-order differential equation,   dCA F (t) F (t) + + k CA = CA,in (t) dt V V where we can identify P(t) = k + (F (t)/V ) and Q(t) = F (t)CA,in (t)/V , and the solution via (6.28) is      F (t) k+ CA(t) = exp − dt V  '      ( F (t) F (t) exp k+ × dt CA,in (t) dt + CA(0) V V

One nonlinear extension of the first-order linear differential equation is to introduce a factor yn to Q(x). This is known as the Bernouli equation, dy + P(x)y = Q(x)yn dx

(6.29)

where n = 1.5 Instead of finding another integrating factor for (6.29), another transformation is used instead. By multiplying (6.29) by (1 − n)y−n and letting z = y1−n , the Bernouli equation is reduced to one that is first-order linear type, that is, dz + (1 − n) P(x)z = (1 − n) Q(x) dx and the solution is given by 8 9 −1 z = φ(x) (1 − n) Q(x)φ(x) dx + C where

 φ(x) = exp

(6.30)

 (1 − n) P(x) dx

6.4 Second-Order Ordinary Differential Equations We will limit our discussion to second-order differential equations that have the form given by   d2 y dy (6.31) = f x, y, dx2 dx In this section, we focus on obtaining transformations that will reduce the order of the equation. Case 1 that follows is likely familiar to most readers, whereas Case 2 invokes similarity transformations to reduce the order. In a later section, we also include the special case of Euler-Cauchy equation, where the transformation converts it to a linear differential equation with constant 5

If n = 1, the Bernouli equation becomes separable.

246

Analytical Solutions of Ordinary Differential Equations

coefficients. The direct solution of linear second-order or higher order equations is deferred to Section 6.5 for cases with constant coefficients or in Chapter 9 for those obtained using Frobenius series methods.

6.4.1 Case 1: Missing Explicit Dependencies on x or y Let p = dy/dx or, equivalently, dy = p dx, then the second derivative d2 y/dx2 can be put in terms of p and x, or in terms of p and y, that is, d2 y dp = 2 dx dx

d2 y dp =p 2 dx dy

or

This will allow the reduction of the second-order differential equation to a first-order differential equation, depending on whether f (x, y, dy/dx) is missing dependencies on x or y, as follows:   d2 y dy x, = f 1 dx2 dx   d2 y dy = f 2 y, dx2 dx

→

→

dp = f 1 (x, p ) dx

(6.32)

dp 1 = f 2 (y, p ) dy p

(6.33)

In either case, a first-order equation will need to be solved, yielding p = S1 (x, C1 ) or p = S2 (y, C1 ), where C1 is an arbitrary constant. With p = dy/dx, the solution is completed by solving another first-order differential equation, that is, dy = S1 (x, C1 ) dx

or

dy = S2 (y, C1 ) dx

which should introduce another arbitrary constant C2 .

The steady-state, one-dimensional concentration profile of substance A under a concentration-dependent diffusivity and flowing through a pipe in the axial direction is described by   d dCA dCA = v αCA dz dz dz

EXAMPLE 6.9.

where α and v are the diffusivity at unit concentration and constant flow velocity, respectively. This can be rearranged to become   d2 CA 1 dCA 1 dCA 2 =β − dz2 CA dz CA dz where β = v/α. This falls under the case where the differential equation is not explicitly dependent on z. With p = dCA/dz, we obtain a first-order linear differential equation given by dp 1 β + p= dCA CA CA

6.4 Second-Order Ordinary Differential Equations

and the solution is

 p



dCA dz

=

=β+

247

m1 CA

where m1 is an arbitrary constant. This is a separable differential equation in CA and z, and the solution is   1 m1 CA − 2 ln βCA + m1 = z + r β β where r is another arbitrary constant. This could be simplified to be q eq = ξ(z) where q = k1 CA − 1 and

ξ(z) = k2 ek1 βz

with k1 and k2 as the new pair of arbitrary constants. Thus an explicit solution for CA(z) is given by   1   CA(z) = W ξ(z) + 1 k1 where W(•) is Lambert’s W-function (also known as the Omega function), defined as the inverse relation of f (w) = w ew , that is, t = qeq ↔ q = W(t)

(6.34)

6.4.2 Case 2: Order Reduction via Similarity Transformations We can extend the definition of similarity transformations, introduced in Definition 6.2 for first-order differential equations, as follows: Definition 6.3. A pair of transformations & x = λα x

;

& y = λβ y

(6.35)

is called a similarity transformation (pair) for a second-order differential equation d2 y =f dx2

 x, y,

dy dx

 (6.36)

if, after substituting (6.35), the new differential equation attains symmetry, given by   d2& y d& y =f & x,& y, d& x2 d& x where λ is the similarity transformation parameter, and α and β are nonzero real constants.

248

Analytical Solutions of Ordinary Differential Equations

If α = 0, then without loss of generality, we could set α = 1, and obtain a new independent variable u and a new dependent variable v given by u

=

v

=

y & y = β xβ & x  1−β  dy  1−β  d& y x = & x dx d& x

(6.37) (6.38)

Using these new variables, the original second-order differential equation can be reduced to a first-order differential equation.6 Let the differential equation (6.36) admit a set of similarity transformations given by (6.35). Using similarity variables u and v defined in (6.37) and (6.38), respectively, the differential equation (6.36) can be reduced to a first-order differential equation given by

THEOREM 6.2.

G(u, v) + (1 − β) v dv = du v − βu where G(u, v) = x

2−β

PROOF.

(6.39)

  dy f x, y, dx

(See Section F.4.2 for proof.)

EXAMPLE 6.10.

Given the second-order differential equation,

 d2 y dy  =x y + x2 − 2y2 2 dx dx we can determine whether a similarity transformation is possible. To do so, let

x4

& x = λα x ,

& y = λβ y

then d2& y d& y d& y = λ−2β& x& y + λ−(β+2α)& x3 − 2λ−2β& y 2 d& x d& x d& x For symmetry, we need β = 2α. Taking α = 1 and β = 2, we have the new variables, x4 λ−(β+2α)&

u=

y x2

and

v=

1 dy x dx

Applying these to (6.39) G(u, v) = uv + v − 2u2 6

→

dv =u du

For the more general case of reducing an n th order ordinary differential equation that admits a similarity transformation, see Exercise E10.8.

6.4 Second-Order Ordinary Differential Equations

249

and the solution is u2 1 dy 1  y 2 +C → = +C 2 x dx 2 x2 where C is an arbitrary integration constant. The last equation is an isobaric type (cf. (6.13)) in which the same combination u = y/x2 will transform the equation to a separable type, that is, v=

2du dx = u2 − 4u + 2C x Solving for u and then for y, the general solution can be simplified to be '  ( y = 2x2 1 + k1 tan k1 ln(k2 x) where k1 and k2 are a new pair of arbitrary constants.

6.4.3 Case 3: Euler-Cauchy Differential Equation The second-order Euler-Cauchy equations are differential equations having the following special form: a2 x2

d2 y dy + a1 x + a0 y = f (x) 2 dx dx

(6.40)

where a0 , a1 , and a2 are constants. These equations can be converted to an associated second-order differential equation that is linear with constant coefficients. Unlike the other two cases, this transformation does not reduce the order of the equation. However, the method for solving linear ordinary differential equations with constant coefficients are well established, and they are discussed in Section 6.5. The key transformation is to set z = ln(x), whose differential is dz = dx/x. This will transform the derivatives with respect to x to derivatives with respect to z as follows:   1 dy d2 y dy 1 dy d2 y = and = 2 − + 2 dx x dz dx2 x dz dz Substituting these into (6.40) yields a2

d2 y dy + (a1 − a2 ) + a0 y = f (ez) 2 dz dz

(6.41)

which is the desired second-order linear equation with constant coefficients. Remarks: 1. The n th -order Euler-Cauchy equations are given by n  i=0

ai xi

di y = f (x) dxi

(6.42)

250

Analytical Solutions of Ordinary Differential Equations

and the same change of variable z = ln(x) will transform it into an nth -order linear differential equation with constant coefficients, involving derivatives with respect to z. 2. Euler-Cauchy equations are special cases of linear differential equations in which the coefficients are analytic; that is, they can be represented by Taylor series. The general approach for these types of differential equations is the Frobenius series solution method, which is covered in Chapter 9. However, the transformation technique described in this section will yield the same solution as the Frobenius series solution, and it has the advantage of being able to immediately determine the character of the solution based on the value of the coefficients a0 , a1 , and a2 . We end this section with a note that there are several other techniques to solve differential equations. We include three of these in Section F.1, namely general Ricatti equations, Legendre transformations, and singular solutions.

6.5 Multiple Differential Equations For the general analysis and solutions of multiple ordinary differential equations, we assume that the system of equations can be cast in the state-space formulation given by d x = f(t, x) dt

(6.43)

where x is an n × 1 column vector that we refer to as the state vector whose element, xi , is known as the ith state variable, ⎞ x1 ⎟ ⎜ x = ⎝ ... ⎠ xn ⎛

(6.44)

and f is an n × 1 column vector of functions, where each f i is in general a nonlinear function of the independent variable t and the state variables, ⎛

⎞ f 1 (t, x1 , . . . , xn ) ⎜ ⎟ .. ⎟ f(t, x) = ⎜ . ⎝ ⎠ f n (t, x1 , . . . , xn )

(6.45)

If f = f (x), then the system of equations is known as autonomous. Note that for these sections, we use t as the independent variable and use x and/or y as the dependent variable.

6.5 Multiple Differential Equations

251

Figure 6.1. System of three simultaneous reversible reaction.

Consider a reactor in which three first-order reactions are occurring simultaneously, as shown in Figure 6.1. The equations for the rate of change of concentrations of components A, B, and C are dCA = (−kABCA + kBACB) + (−kACCA + kCACC) dt dCB = (−kBCCB + kCBCC) + (−kBACB + kABCA) dt dCC = (−kCACC + kACCA) + (−kCBCC + kBCCB) dt These equations can now be formulated in state-space form EXAMPLE 6.11.

dC =KC dt where, ⎛

⎞ CA C = ⎝ CB ⎠ , CC

⎛

−(kAB + kAC) ⎝ K= kAB kAC

kBA −(kBA + kBC) kBC

⎞ kCA ⎠ kCB −(kCA + kCB)

Aside from a set of first-order differential equations that are already of the form given in (6.43), higher order equations can also cast in state-space forms. Consider a high-order differential equation in which the highest order derivative can be explicitly written as follows: dn y =f dtn

  dy dn−1 y t, y, , . . . , n−1 dt dt

(6.46)

By assigning states to each of y and its derivatives all the way to (n − 1)th derivative, x1 = y,

x2 =

dy , dt

···

xn =

dn−1 y dtn−1

(6.46) can then be written as ⎛

d dt

x1 .. .

⎜ ⎜ ⎜ ⎝ xn−1 xn

⎞

⎛

⎟ ⎜ ⎟ ⎜ ⎟=⎜ ⎠ ⎝

x2 .. . xn f (t, x1 , . . . , xn )

⎞ ⎟ ⎟ ⎟ ⎠

(6.47)

252

Analytical Solutions of Ordinary Differential Equations EXAMPLE 6.12.

Consider Van der Pol’s equation given by d2 y dy + (y2 − b) + y = 0 dt2 dt

Let x1 = y and x2 =

dy , then dt dx1 = dt dx2 = dt

or

x2 x2 (b − x21 ) − x1

⎛ d x = f(x) = ⎝ dt

⎞

x2

⎠

bx2 − x21 x2 − x1

The state-space formulation is not only helpful in obtaining analytical solutions. It is in fact the standard form used in numerical solutions of high-order differential equations, as is discussed in Chapter 7.

6.5.1 System of Linear Equations We now limit the discussion to multiple differential equations that are linear. Let the functions f i be a linear combination of the state variables x j , j = 1, . . . , n, and a forcing functions bi (t), that is, f i (t, x1 , . . . , xn ) = ai1 (t)xi + · · · ain (t)xn + bi (t) Then equation (6.43) can be written in matrix form as d x = A(t)x + b(t) dt

(6.48)

where ⎛

a11 (t) ⎜ .. A=⎝ . an1 (t)

··· .. . ···

⎞ a1n (t) ⎟ .. ⎠ . ann (t)

⎛

;

⎞ b1 (t) ⎜ ⎟ b(t) = ⎝ ... ⎠ bn (t)

In the next two sections, we solve (6.48) by the introduction of matrix exponentials when A is constant. The details for the analytical solution are given in Section 6.5.2. A concise formula for the explicit solution for the case with constant A is given in (F.21). When A = A(t), the matrix exponentials are generalized to matrizants. The solutions for these cases are difficult to generalize. However, two special cases are

6.5 Multiple Differential Equations

253

considered. One case is when A(t) and A(τ) commutes. The other case is when A(t) can be represented by power series in t.

6.5.2 Matrix Exponentials For constant A, the solution involves the use of the matrix exponential, eAt . We can apply (3.34) from Section 3.7 eAt = I + tA +

t2 2 t3 3 A + A + ··· 2! 3!

(6.49)

Some of the properties of eAt are given by the following theorem: Let A and W be square matrices of the same size and let t and s be scalars. Then eAt satisfies the following properties:

THEOREM 6.3.

(i) eAs eAt = eA(s+t)  −1 (ii) eAt = e−At (iii) eAt eWt = e(A+W)t (iv) PROOF.

(6.50) (6.51) if and only if AW = WA

d At e = AeAt = eAt A dt

(6.52) (6.53)

(See Section F.4.3 for proof.)

Using Theorem 6.3, with x = x(t),       d  −At  d −At d d −At −At −At e x = x+e e x = −e Ax + e x dt dt dt dt   d = e−At x − Ax (6.54) dt Rearranging (6.48) and premultiplying by e−At ,   d e−At x − Ax = e−At b (t) dt then with (6.54), d −At e x = e−At b (t) dt Integrating from 0 to t, with eA0 = I, −At

e

 x(t) − x(0)

t

=

e−Aτ b (t) dτ

0

x(t)

=



t

eAt x(0) +

eA(t−τ) b (τ)dτ 0

(6.55)

254

Analytical Solutions of Ordinary Differential Equations

If matrix A is diagonalizable (cf. Section 3.5), with A = VV −1 , (6.55) can be simplified to be x(t)



t

=

Vet V −1 x(0) + Vet V −1

Ve−τ V −1 b (τ)dτ

=

   t e−τ V −1 b (τ)dτ Vet V −1 x(0) +

0

(6.56)

0

EXAMPLE 6.13.

Given the differential equation, dx = Ax + b(t) dt

with

⎛

−3 ⎝ A= −2 2

2 1 −2

⎞ 1 2 ⎠ −4

⎛

⎞ 4 − e−t ⎠ b(t) = ⎝ 2 −2t 1+e ⎞

; ⎛

1 x(0) = ⎝ 0 ⎠. 1 The eigenvalues of A are (−3, −2, −1), and A is diagonalizable. Thus A = VV −1 , with ⎛ ⎞ ⎛ ⎞ 0 1 1 −3 0 0 V = ⎝ −1 0 1 ⎠ ;  = ⎝ 0 −2 0 ⎠ 2 1 0 0 0 −1

subject to the initial condition:

Let

⎛  q

= 0

r

=

s

=

t

⎜ ⎜ e−τ V −1 b(τ)dτ = ⎜ ⎝

− 67 + et + 21 e2t − 31 e3t 1 2

− t − 2et + 23 e2t

t + et − e−t   ⎛ 5 + (t − 2) e−t + −t − 21 e−2t 2 ⎜   ⎜ Vet q = ⎜ 34 + t − 21 e−t − 2e−2t + 67 e−3t ⎝   5 − e−t + −t + 25 e−2t − 37 e−3t 6 ⎛ −2t ⎞ e Vet V −1 x(0) = ⎝ 0 ⎠ e−2t

The solution is then given by ⎛ ⎜ ⎜ x=r+s=⎜ ⎝

  + (t − 2) e−t + −t + 21 e−2t   4 + t − 21 e−t − 2e−2t + 67 e−3t 3   5 − e−t + −t + 27 e−2t − 37 e−3t 6 5 2

⎞ ⎟ ⎟ ⎟ ⎠

⎞ ⎟ ⎟ ⎟ ⎠

⎞ ⎟ ⎟ ⎟ ⎠

6.5 Multiple Differential Equations

If A is not diagonalizable, we could use the Jordan canonical form. However, a more efficient alternative is to use the method of finite sums (cf. Section 3.7, Case 3) for evaluating eAt .

EXAMPLE 6.14.

Given the differential equation dx = Ax dt

with

⎛

−2

⎜ ⎜ 1 ⎜ A = ⎜ −2 ⎜ ⎝ 1

0 −2

0 −

1 2

2

0

⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠

The eigenvalues of A are λ = (−2, −1, −1). Following the method discussed in Section 3.7, the Cayley-Hamilton theorem suggests that there exist scalar functions c2 , c1 , and c0 such that7 eAt = c2 t2 A2 + c1 tA + c0 I To determine these unknown coefficients, we need three independent equations. The first two equations are obtained by applying the equation given by eλt = c2 t2 λ2 + c1 tλ + c0 to each distinct eigenvalue, λ = −2 and λ = −1, that is, e−2t

=

4c2 t2 − 2c1 t + c0

e−t

=

c2 t2 − c1 t + c0

The third equation is obtained by taking the derivative, with respect to λ, of both sides of equation (6.14) and then setting λ = −1 (because this is the repeated root), te−t = −2c2 t2 + c1 t Combining all three equations, we solve for c2 t2 , c1 t, and c0 instead of c2 , c1 , and c0 . This saves us from having to invert a nonconstant matrix, and it also adds to the efficiency in the succeeding steps. Thus ⎛ ⎞⎛ ⎞ ⎛ −2t ⎞ 4 −2 1 c2 t2 e ⎝ 1 −1 1 ⎠ ⎝ c1 t ⎠ = ⎝ e−t ⎠ te−t −2 1 0 c0 and

7

⎛

⎞ ⎛ ⎞ c2 t2 (t − 1) e−t + e−2t ⎝ c1 t ⎠ = ⎝ (3t − 2) e−t + 2e−2t ⎠ 2te−t + e−2t c0

In Section 3.7 Case 3, these scalars are constants. However, in applying those same methods here, we take the variable t as a parameter to apply Cayley-Hamilton’s theorem. This will necessarily result in the coefficients ci to be functions of t.

255

256

Analytical Solutions of Ordinary Differential Equations

Next, we apply these coefficients to obtain eAt , eAt

=

=

c2 t2 A2 + c1 tA + c0 I ⎛ e−2t 0 ⎜ ⎜ 1  −2t  −t ⎜ (−t + 1) e−t ⎜ 2 e −e ⎝  −2t  −e + e−t 2te−t

and the solution is then given by ⎛ e−2t ⎜ ⎜ 1  −2t  −t x=⎜ ⎜ 2 e −e ⎝  −2t  −e + e−t

⎞

0

(t + 1) e−t

0

⎞

0 −t

⎟ ⎟ ⎟ x0 ⎟ ⎠

− 2t e−t

(−t + 1) e 2te−t

⎟ ⎟ ⎟ ⎟ ⎠

− 2t e−t

(t + 1) e−t

where x0 = x(0) is the vector of initial conditions.

A general formulation of the finite series method, including details, is given in Section F.3 as an appendix.

6.5.3 Matrizants Consider the linear matrix differential equation given by d x = A(t)x + b(t) dt

(6.57)

subject to initial condition: x(0) = x0 Let us first consider the solution of the homogenous equation, that is, with b(t) = 0, d x = A(t)x dt

(6.58)

One solution approach is Picard’s method. It begins by integrating the differential equation,  t dx = A(t)xdt → x(t) = x0 + (6.59) (A(τ1 )x(τ1 )) dτ1 0

After recursive application of (6.59),   t  t  A(τ1 ) x0 + (A(τ1 )x(τ1 )) dτ1 = 0



A(τ2 )x(τ2 )dτ2 dτ1

0

0

=



τ1

t



A(τ1 )dτ1 x0 +

0



t

τ1

A(τ1 ) 0

(A(τ2 )x(τ2 )) dτ2 dτ1 0

Let Qk be defined as 

t

Q1 (t) =

A(τ1 )dτ1 0

for k = 1

(6.60)

6.5 Multiple Differential Equations

257

and 



t

Qk (t) = 0



τ1

A(τ1 )

A(τ2 ) 0

τ2



τk−1

A(τ3 ) · · ·

0

A(τk )dτk · · · dτ3 dτ2 dτ1

(6.61)

0

for k > 1. Assuming convergence, the solution becomes an infinite series given by ∞   Qk x0 = M(t)x0 (6.62) x(t) = x0 + k=1

where M(t) = I +

∞ 

Qk (t)

(6.63)

k=1

M(t) is known as the matrizant or fundamental matrix of the differential equation given in (6.58). It has the following properties: M(0) = I

and

dM = A(t)M(t) dt

(6.64)

Let the elements of the state matrix A be bounded; then the matrizant M defined by (6.63), with Qk defined in (6.61), is invertible.

THEOREM 6.4.

PROOF.

(See Section F.4.4 for proof.)

Corresponding to (6.64), we have M−1 (0) = I

and

dM−1 /dt = −M−1 (t)A(t)

(6.65)

To show this, d  −1  d  −1  d M M = M M + M−1 (M) dt dt dt     d d M−1 + M−1 (M) M−1 dt dt d  −1  M + M−1 AMM−1 dt d  −1  M dt

=

0

=

0

=

0

=

− M−1 A

Applying (6.65), we can establish the following identity,       d d  −1  d −1 d −1 −1 −1 M x x+M = M x = −M Ax + M x dt dt dt dt   d −1 x − Ax (6.66) = M dt

258

Analytical Solutions of Ordinary Differential Equations

We can now solve the nonhomogeneous case, that is, b(t) = 0, d x − A(t)x dt   d −1 M x − Ax dt d  −1  M x dt   d M−1 x M−1 (t)x(t) − M−1 (0)x(0)

=

b(t)

=

M−1 b(t)

=

M−1 b(t)

=

M−1 b(t)dt  t M−1 (τ)b(τ)dτ

= 

x(t) = M(t)x(0) + M(t)

0 t

M−1 (τ)b(τ)dτ

(6.67)

0

where we used the fact that M(0) = M−1 (0) = I. Equation (6.67) is the general solution of the linear nonhomogeneous equation.8 However, matrizants M are generally difficult to evaluate. One important case that is easier to solve is when matrices A(t) and A(τ) commute. If A(t)A(τ) = A(τ)A(t) for τ = t, then A and Qk will also commute. Thus  where Q1 =

M(t) = eQ1

and

M−1 (t) = e(−Q1 )

(6.68)

t

A(τ1 )dτ1 . (See Exercise E6.19 for an example of this case.) When A 0

is constant, the matrizant is simply given by M = eAt as before. We use matrizants later to help us solve linear boundary value problems via the shooting method in Section 7.5. However, in that section, we just use numerical initial value problem (IVP) solvers developed in Chapter 7 to evaluate the matrizants at the terminal point T .

6.6 Decoupled System Descriptions via Diagonalization If the system matrix A is constant and diagonalizable, a change of coordinates can yield a decoupled set of differential equations. Let A = VV −1 , where  is the diagonal matrix of eigenvalues, and V is the matrix containing the eigenvectors of A. Now let z be the new set of state variables defined by ⎞ ⎛ z1 ⎟ ⎜ (6.69) z = ⎝ ... ⎠ = V −1 x zn where x is the original set of state variables, then ⎛ d d z = V −1 x dt dt 8

and

⎞ zo,1 ⎟ ⎜ zo = ⎝ ... ⎠ = V −1 xo zo,n

This solution approach, where the solution of the linear homogeneous case b(t) = 0 can be used to extend the solution to the linear nonhomogeneous case b(t) = 0, is also known as Duhamel’s principle.

6.6 Decoupled System Descriptions via Diagonalization

259

The original linear equation can be transformed as follows: d x dt d V −1 x dt d z dt where

=

Ax + b(t) = VV −1 x + b(t)

=

V −1 x + V −1 b(t)

=

z + Q(t)

(6.70)

⎞ q1 (t) ⎟ ⎜ Q(t) = V −1 b(t) = ⎝ ... ⎠ qn (t) ⎛

Because  is diagonal, (6.70) can be rewritten as dz1 dt

=

λ1 z1 + q1 (t)

.. . dzn dt

=

(6.71) λn zn + qn (t)

which is a set of decoupled differential equation. Each decoupled differential equation is a first-order linear differential equation whose solution is given by  t zk (t) = eλk t zo,k + eλk (t−τ) q(τ)dτ (6.72) 0

The original state variables, x, can be recovered using (6.69), x = Vz

(6.73)

which is the same solution given in (6.56). Under the decoupled system description, the solution given in (6.72) can be used to assess the stability of the solution. If any of the eigenvalues, say, λk , have positive real parts, the value of zk in (6.72) will keep increasing unboundedly with increasing values of t. Furthermore, because x is a linear combination of the elements of z, the same stability conclusion is the same for x(t). This conclusion is also true for the non-diagonalizable case, as stated in the following theorem: THEOREM 6.5.

For the linear system described by d x = Ax + b(t) dt

(6.74)

with A constant and b(t) bounded, then x(t) is unstable if any of the eigenvalues of A has a positive real part. PROOF.

(See Section F.4.5 for proof.)

260

Analytical Solutions of Ordinary Differential Equations

Figure 6.2. System of three simultaneous reversible reactions.

When decoupling is possible, the trajectories will move along straight lines. In the following example, this fact has been used to solve a parameter estimation problem in which strong interaction among the original set of variables is present. Wei Prater Kinetics Wei and Prater9 used the idea of decoupling to obtain kinetic parameters of simultaneous reversible first-order reactions of N chemical components. For three components undergoing first-order kinetics as shown in Figure 6.2, the system is described by ⎞ ⎛ ⎞ ⎛ ⎛ ⎞ xA −(kAB + kAC) kBA kCA x d ⎝ A⎠ ⎠ ⎝ xB ⎠ xB = Kx = ⎝ kAB −(kBA + kBC) kCB dt xC kAC kBC −(kCA + kCB) xC

EXAMPLE 6.15.

where xi denotes mass fraction and xA + xB + xC = 1. In matrix form, d x = Kx dt where ⎛

⎞ xA x = ⎝ xB ⎠ xc

⎛

− (kAB + kAC) K=⎝ kAB kAC

kBA − (kBA + kBC) kBC

⎞ kCA ⎠ kCB − (kCA + kCB)

The objective is to estimate the six kinetic coefficients, kij , using experimental data. A typical graph is shown in Figure 6.3. Two things are worth noting in Figure 6.3: (1) all curves converge to a single point, and (2) there are two straight-line reactions. The main challenge in this parameter estimation problem is to determine the kinetic coefficients when no experiment exists that isolates dependence on only one component at a time. Wei and Prater decided to look for an alternative set of coordinates such that under new coordinates, the pseudoconcentrations are decoupled. Thus let us define a set of pseudo-concentration variables as follows: ⎞ ⎛ y1 y = ⎝ y2 ⎠ = V −1 x y3 where V is the matrix of the eigenvectors of K.10 This should result in a decoupled system dy = y dt 9

Wei, J. and Prater, C. D., The Structure and Analysis of Complex Reaction Systems, Advances in Catalysis, 13, Academic Press, New York (1962). 10 This naturally assumes that K is diagonalizable.

6.6 Decoupled System Descriptions via Diagonalization

261

Figure 6.3. Experimentally observed compositions for butene isomerization. (Data adapted from Froment and Bischoff, Chemical Reactor Analysis and Design, J. Wiley and Sons, 1979, p. 21).

where  is the diagonal matrix of eigenvalues of K. The solution of the decoupled system is given by y1 = y1 (0)eλ1 t

,

y2 = y2 (0)eλ2 t

and

y3 = y3 (0)eλ3 t

In terms of the original concentration, this becomes x = Vy

=

y1 v1 + y2 v2 + y3 v3

=

y1 (0)eλ1 t v1 + y2 (0)eλ2 t v2 + y3 (0)eλ3 t v3

(6.75)

Note that det(K) = 0 because the last row is the negative of the sum of the upper two rows. Recall from one of the properties of eigenvalues (cf. Property 7 of Section 3.3) that the product of the eigenvalues of matrix K is equal to the determinant of K. Because K is singular, then at least one of the eigenvalues of K must be zero. Without loss of generality, set λ1 to zero. Then the corresponding eigenvector, v1 , behaves according to definition of eigenvectors, Kv1 = λ1 v1 = 0 Thus, had the experiment started at x∗ = y1 (0)eλ1 t v1 , d ∗ x = Kx∗ = 0 dt which means x∗ is the equilibrium point of the process, xeq , that is, xeq = y1 (0)v1 Let us now look at the deviations from equilibrium: x − xeq . From (6.75), x − xeq = y2 (0)eλ2 t v2 + y3 (0)eλ3 t v3 which is a linear combination of vectors v2 and v3 .

(6.76)

262

Analytical Solutions of Ordinary Differential Equations B

v2 r A

v3

xeq = v 1 v2

s

v3

q

C

Figure 6.4. Using straight-line reactions to identify v2 and v3 .

If we start the process at an initial point where y2 (0) > 0 and y3 (0) = 0, we obtain a reaction path that follows the direction of v2 , that is, a straight-line path. Along this path, only the coefficient given by y2 (0)eλ2 t decreases with time. The eigenvalue λ2 can then be found using the least-squares method by estimating the slope of the linear equation ln(x − xeq ) = ln(y2 (0)v2 ) + λ2 t as we follow the path along v2 . Using the other straight-line reaction, we can obtain λ3 by starting at y3 (0) > 0 and y2 (0) = 0, which will be a path along v3 . Thus the eigenvalue λ3 can be found in a similar manner, using the least-squares method to estimate the slope of the linear equation ln(x − xeq ) = ln(y3 (0)v3 ) + λ3 t Eigenvectors v2 and v3 can be obtained directly from the data, as shown in Figure 6.4, whereas the equilibrium point v1 = xeq can be read off the plot (point r). By subtracting the mass fractions at the start of one of the straight lines (point s in the figure) and the equilibrium point (point r), the resulting vector can be designated as v2 . Likewise, using the other straight-line reaction that is not along the previous line, one could also subtract the start point (point q) and the end point (point r) to determine v3 . Combining all the results so far: v1 = xeq , v2 , v3 , λ1 = 0, λ2 and λ3 , we can build matrices V and , ⎞ ⎛ 0 0 0  = ⎝ 0 λ2 0 ⎠ V = (xeq |v2 |v3 ) 0 0 λ3 Finally, matrix K can be reconstructed as follows: K = VV −1

6.7 Laplace Transform Methods For a multiple first-order linear differential equations, where the state matrix A is constant, that is, d x = Ax + b(t) dt

(6.77)

6.8 Exercises

263

the method of Laplace transforms can be used to transform the set of differential equations into a set of algebraic equations. The Laplace transform of a function f (t) is defined as  ∞ f (t)e−st dt (6.78) L [ f (t)] = 0

where s is the Laplace transform variable, which spans the right-half complex plane. Details, including several properties of Laplace transforms, can be found in Section 12.4. In our case, we use the property of derivatives given by ' ( df L = sL [ f (t)] − f (0) (6.79) dt and the convolution theorem L [ f ∗ g] = L [ f ] L [g] where the convolution of f (t) and g(t) is defined by  t f ∗g = f (t − τ) g (τ) dτ

(6.80)

(6.81)

0

For the special case of f = eAt , recall (6.53) and apply (6.79), ' (   d At L e = L AeAt dt    At  s L e − I = AL eAt or

−1    L eAt = sI − A

(6.82)

Applying (6.79) and (6.82) to (6.77), sI L [x] − x(0)

=

L [x]

=

AL [x] + L [b]  −1   sI − A x(0) + L [b]

  Next, use the inverse Laplace transform, L−1 [·] defined by L−1 L [ f ] = f ,         x = L−1 L eAt x(0) + L−1 L eAt L [b]   = eAt x(0) + eAt ∗ (b)  ∞ eA(t−τ) b(τ)dτ = eAt x(0) + 0

which is the same solution given by (6.55). 6.8 EXERCISES

E6.1. Consider the differential equation,     df S(y) dx + Q(y) f (x) + R(y) dy = 0 dx 1. Find the integrating factor using either (6.24) or (6.25).

264

Analytical Solutions of Ordinary Differential Equations

2. Apply the result to solve the following differential equation: (ydx + dy) eαx + R(y)dy = 0 E6.2. The axial flow along the z-direction of an incompressible fluid through a circular tube of radius R is given by the following differential equation:   d dvz r = βr dr dr where β=

PL − P0 μL

Obtain vz(r) under the conditions that vz is finite at r = 0 and vz = 0 at r = R. E6.3. The steady-state model for the biodegradation of substance A in a spherical pellet is given by d2 CA 2 dCA + = αC2A dr2 r dr where α = k/DAB is constant. Use the similarity transformation to reduce the order of the equation. E6.4. Obtain the general solution for the following differential equations: dy 4x + 5y + 2 1. = dx x+y−3 2.

dy 3x − y + 3 = dx y − 3x + 2

3.

dy = (2x + y) (2x + y + 4) dx

E6.5. Find the integrating factor for the following equation:     a b sin (ax + by) dx + sin (ax + by) + 2 dy = 0 y y E6.6. Consider a thin cylindrical wire of length L and diameter D exposed to a constant surrounding temperature T a , while the ends are fixed to be T 0 and T L at x = 0 and x = L, respectively. The temperature is assumed to be uniform at any cross-section and the temperature profile along the length of the wire is described by the following equation resulting from the energy balance:11 ' ( d dT k(T ) = β (T − T a ) dx dx where β = πDU/A, with D, A, and U as diameter, cross-sectional area, and heat transfer coefficient, respectively. Suppose the material has a temperature-dependent conductivity given by k(T ) = k0 − αT 11

This problem is adapted from an example given in Jenson and Jeffreys, Mathematical Methods in Chemical Engineering, 2nd Ed., Academic Press, 1977.

6.8 Exercises

265

1. Show that the relationship between x and T is given by  T (x) km − α (T − T a ) " x= dT 1 T (0) C + 3 β (T − T a )2 (−2α (T − T a ) + 3km ) where C is an arbitrary constant and km = k0 − αT a . W 2. Let the parameters be given by α = 0.0067 cm(Wo C)2 , k0 = 1.5 cm· oC , Ta = 30o C, T 0 = 60o C, T L = 100o C, L = 12cm and β = 0.012 cmW 3 ·o C . Determine the value of C. E6.7. The growth of microbial colony is modeled by dN = kN − βN2 dt where N is the population of the colony of bacteria, k > 0 is the specific growth rate, and β > 0 is the death rate coefficient. 1. Let k and β be constant. Obtain the solution with N(0) = N0 . The solution is called the logistic solution. 2. Determine the equilibrium value, Neq , of the logistic solution, in terms of k and β. Neq 3. Plot the logistic solution for 0 < No < and show that a lag will become 2 more prominent as No is closer to zero. 4. Solve the differential equation when β = β(t), where k < β < βmax . E6.8. Generate a first-order differential equation that admits a similarity transformation and yet is not an isobaric differential equation. E6.9. Consider the Nth order Euler-Cauchy differential equation:  N   dk y ak xk k = f (x) dx k=0

1. Using the change of variable z = ln(x), show that the kth derivatives dk y/dxk can be put in terms of the derivatives dk y/dzk as follows: dy dx

=

d2 y dx2

=

1 dy x dz   1 dy d2 y − + x2 dz dx2

.. . dk y dxk

=

k 1  d y b k, xk dz =1

.. . where bk,

⎧ ⎨ − (k − 1) bk−1,1 = bk−1,−1 − (k − 1) bk−1, ⎩ 1

if  = 1 if 1 <  < k if  = k

266

Analytical Solutions of Ordinary Differential Equations

2. Solve the following third-order Euler-Cauchy equation: x3

d3 y d2 y dy 1 + x2 2 + x +y= 3 dx dx dx 2x

E6.10. Let A and B given by  −1 A= 0

1 −1



 B=

−1 1

0 −1



Show that because A and B is not commutative, we have eAt eBt = e(A+B)t E6.11. Let n be an integer. Find the value of k such that the following equation admits a similarity transformation: dy = yk f (xyn ) dx Solve the following equation using a similarity transformation: 1 dy = xy + 1 y2 dx E6.12. A guided missile is fired at a target that is moving at a constant speed sT . Let the missile also travel at a constant speed sm > sT while being aimed at the target (see Figure 6.5), with dx = k(t) (xT − x) dt dr = k(t) (rT − r) → dy dt = k(t) (yT − y) dt T

T

target

Figure 6.5. The tangent of the path of a guided missile is directed at the target at all times.

guided missile (x,y)

where r and rT are the position vectors of the missile and target, respectively, and k is a proportionality function. After dividing dx/dt by dy/dt, and then taking another time derivative followed by one more division by dy/dt, show that we can end up with "   1 + (dx/dy)2 dx dyT d2 x dxT − y) + − =0 (y T dy2 sm dy dt dt ! where we used the fact that sm = (dx/dt)2 + (dy/dt)2 . Consider the simple case in which the target follows a straight path with yT (t) = YT 0 and xT (t) = X T 0 + sT t, where YT 0 and X T 0 are constants, and initial conditions x(0) = 0 and y(0) = 0. Find the relationship between x and y, also known as the pursuit path.12 12

Problem based on Burghes, D. N. and Borrie, M. S. Modeling with Differential Equations, Ellis Horword, Ltd, 1981.

6.8 Exercises

267

E6.13. For the Ricatti equation given by dy β γ = y2 + y + 2 dx x x Obtain the general solution if (1 + β)2 > 4γ. (See Section F.1.1 for an approach to solving Ricatti equations.) E6.14. Given the set of differential equations, ⎛ −3 2 ⎜ 0 −2 d x=⎜ ⎝ 1 −1 dt 1 −1

−4 1 0 2

⎞ 2 −1 ⎟ ⎟x 0 ⎠ −2

Obtain the solution when the initial condition is given by xT =  1 1 0 0 . E6.15. Let A[=]2 × 2, such that α=

trace(A) 2

" β=

and

Show that exp (At) =

trace(A)2 − 4 det(A) 2

= 0

 eαt  p (t)I + q(t)A β

where p (t) = β cosh (βt) − α sinh (βt)

and

q(t) = sinh (βt)

Thus show that the solution of d x = Ax + b(t) x(0) = x0 (6.83) dt is given by  t α(t−τ)    eαt  e x(t) = p (t)I + q(t)A x0 + p (t − τ)I + q(t − τ)A b(τ)dτ β β 0 Also, using l’Hospital rule, show that when β = 0,   exp(At) = eαt (1 − αt) I + At and thus the solution to (6.83) when β = 0 becomes  t     eα(t−τ) [1 − α(t − τ)] I + A [t − τ] b(τ)dτ x(t) = eαt (1 − αt)I + At x0 + 0

E6.16. Consider an Nth -order linear differential equation, dN dN−1 dy y + αN−1 N−1 y + · · · + α1 + α0 y = u (t) N dt dt dt 1. By defining N variables xi as di−1 y i = 1, . . . , N dti−1 order differential equation can be put in state-space xi =

show that the Nth form

d x = Ax + b(t) dt

268

Analytical Solutions of Ordinary Differential Equations

where A is in the companion form (cf. E3.8), ⎛ 0 1 0 ··· ⎜ 0 0 1 ··· ⎜ ⎜ .. . . .. .. .. A=⎜ . . ⎜ ⎝ 0 0 0 ··· −α0 −α1 −α2 · · ·

0 0 .. . 1 −αN−1

⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠

with y = x1 . 2. From E3.8, if the eigenvalues are distinct, the matrix of eigenvectors V , for the companion matrix A, is given by the Vandermonde matrix, ⎛ ⎞ 1 1 ⎜ .. ⎟ V = ⎝ ... ··· . ⎠ N−1 N−1 λ1 λN whose inverse is given by V −1 = where 



 hij

hij

=

ωi,j −1 βi

(λ − λ j )

=

ωi,N−1 λN−1 + · · · + ωi,1 λ + ωi,0

βi

=

j =i



(λi − λ j )

j =i

Using these formulas, find the solution of d3 y d2 y dy + 5 + 7.75 + 3.75y = 3 + e−2t 3 2 dt dt dt

y(0) = 1,

dy d2 y (0) = 2 (0) = 0 dt dt

using matrix exponentials. E6.17. Table 6.1 gives data for the set of reversible reactions given in Example 6.15. Find the values of kinetic rate constants kAB, kBA, kBC, kCB, kCA, and kAC.13 E6.18. Use the Legendre transformation (see Section F.1.2) to solve the following differential equation:  2 dy + a = ax + y dx E6.19. Let the system matrix A be time-varying, given by A (t) = f (t)A1 + g(t)A2 where A1 and A2 are constant matrices. 1. Show that A(t) and A(τ) commutes if A1 and A2 commutes. Generalize this for the case where N  hk (t)Ak A(t) = k=1 13

A set of MATLAB programs are available on the book’s webpage for obtaining ternary plots. See the document plotTernaryTutorial.pdf for instructions.

6.8 Exercises

269

Table 6.1. Data sets for Wei-Prater kinetics Data set 1

Data set 1

Data set 1

Time

CA

CB

Time

CA

CB

Time

CA

CB

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 7.0 9.0 11.0 12.0

0.900 0.744 0.629 0.543 0.480 0.433 0.399 0.373 0.354 0.340 0.329 0.309 0.302 0.300 0.300

0.000 0.077 0.135 0.178 0.209 0.233 0.250 0.263 0.272 0.279 0.285 0.295 0.298 0.299 0.299

0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 10.0 11.0 13.0 14.0 18.0 21.0

0.100 0.220 0.279 0.305 0.315 0.317 0.316 0.314 0.311 0.307 0.305 0.303 0.302 0.300 0.300

0.000 0.043 0.090 0.134 0.171 0.201 0.225 0.243 0.257 0.276 0.282 0.290 0.292 0.297 0.299

0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 10.0 11.0 13.0 15.0 21.0

0.100 0.152 0.190 0.219 0.240 0.255 0.267 0.276 0.282 0.287 0.290 0.293 0.296 0.298 0.300

0.900 0.745 0.629 0.544 0.481 0.434 0.399 0.373 0.354 0.340 0.330 0.322 0.312 0.307 0.301



 2 −2

2. Let

1 A1 = −1

 A2 =

1

1

− 12

− 12



and f (t) = −e−2t

g(t) = −2e−5t

Obtain the solution of the following set of differential equations using matrizants:   d x = f (t)A1 + g(t)A2 x dt   subject to the initial condition, xT (0) = 1 −0.6 . E6.20. An Emden-Fowler equation is given by d2 y a dy + + byn = 0 dx2 x dx Show that for n = 1, a similarity transformation can reduce the equation to a first-order differential equation. E6.21. The relationships between current i(t) and voltage v(t) across inductors, capacitors, and resistors are given by v

=

Ri

dv dt

=

v

=

1 i C di L dt

Resistors Capacitors Inductors

A simple third-order Butterworth filter based on the Cauer topology using two inductors, one capacitor, and one resistor is shown in Figure 6.6. Using

270

Analytical Solutions of Ordinary Differential Equations L1

V

in

L2

C i 2

i1

R

V

out

Figure 6.6. A Butterworth filter using a Cauer Topology.

Kirchhoff’s laws, we end up with the following differential equations: 1 d 2 i1 (i1 − i2 ) + L1 2 C dt

=

dvin dt

1 d 2 i2 di2 (i2 − i1 ) + L2 2 + R C dt dt

=

0

where i1 and i2 are the currents through inductors L1 and L2 , respectively, and vin (t) is a given input voltage. 1. Obtain the state-space formulation based on the state vector defined by (x1 , x2 , x3 , x4 )T = (i1 , di1 /dt, i2 , di2 /dt). 2. Determine the characteristic function for this linear system. For R = 1 choose values of L1 , L2 , and C so that the characteristic equation reduces to   s s3 + 2s2 + 2s + 1 = 0 3. Solve for vout = i2 (t)R using the following parameters and initial conditions: L1 = 1.5, L2 = 0.5, C = 4/3, R = 1, vin (t) = sin(0.2t) − 0.1 sin(5t), and x = 0. Plot vout (t) vs. t together with vin (t) vs. t to see the performance of the Butterworth filter for this example. E6.22. Show that the complete solution of 

dy −5 dx

2 + 5x = y + 2

is given by '

1 y(x) = (x − 5 + C) 2

(2 + 5x − 2

where C is arbitrary. Find the singular solution for this differential equation and plot the singular solution together with the complete solution, that is, using different values for C, thus verify that the region where the solution exists is determined by an envelope that is obtained from the singular solution. E6.23. A spring and dashpot system shown in Figure 6.7 is often used to model the dynamics of mechanical systems in motion. The force balance yields a linear second-order system given by m

dx d2 x + C + kx = F (t) 2 dt dt

6.8 Exercises

271 s

F

Figure 6.7. A spring and dashpot system.

m dx dt

d

1. Obtain a state-space model for a system of N spring and dashpots as shown in Figure 6.8. kN

k1

mN

Figure 6.8. A series of springs and dashpots system.

m2

cN

m1 c1

xN

x2

x1

2. Let p 0 (s) = 1 and p 1 (s) = m1 s2 + C1 s + k1 . Then it can be shown that the characteristic equation for this system is given by p N (s)/( N j =1 m j ) = 0 for N ≥ 1, where  2   p  = m s2 + C s + k p −1 − C−1 s + k−1 p −2 ;  > 1 Verify this to be the case for N = 2. 3. Let N = 3 and F = 0.25 tanh(t − 5) together with the following parameters i

mi

ki

Ci

1 2 3

10 8 12

5 5 10

0.1 0.2 0.1

Will the system be stable or unstable? E6.24. An equilibrium reaction from A to B is given by 2A ⇔ B The kinetics are given by 1 dCA 2 dt dCB dt

=

kbCB − k f C2A

=

k f C2A − kbCB

1. Show that the equations can be combined to yield d2 CA dCA + (4k f CA + kb) =0 2 dt dt 2. Solve for CA(t) and CB(t) with the initial conditions CA(0) = CA0 and CB(0) = CB0 .

F

272

Analytical Solutions of Ordinary Differential Equations

E6.25. Let CR be the RNA concentration and CE as the enzyme concentration. One model for protein synthesis is given by dCR = r(CE ) − k1 CR dt dCE = k2 CR − k3 CE dt where r(CE ) is the rate of production of protein based solely on the enzyme concentration, and k1 , k2 and k3 are constants. 1. Show that this can be combined to yield a second-order reaction given by d2 CE dCE +α = k2 CE r(CE ) − βCE 2 dt dt 2. Assuming Michaelis-Menten kinetics for r(CE ), that is, k4 CE r(CE ) = k5 + CE Reduce the equation into a first-order differential equation with dependent variable p = dCE /dt and independent variable z = CE . E6.26. Prove that if A(t)A(τ) is commutative, then with Q1 (t) and Qk (t) as defined in (6.60) and (6.61), respectively, becomes 1 Qk (t) = Q1 (t)k k! and thus showing that the matrizant formula given in (6.68) M(t) = eQ1 (t) is valid.

7

Numerical Solution of Initial and Boundary Value Problems

In several cases, the analytical solution of ordinary differential equations, including high-order, multiple, and nonlinear types, may not be easy to obtain or evaluate. In some cases, it requires truncation of an infinite series, whereas in other cases, it may require numerical integration via quadratures. An alternative approach is to determine the solution directly by numerical methods. This means that the solution to a differential equation will not be given as a function of the independent variables. Instead, the numerical solution is a set of points discretized over the chosen range of the independent variables. These points can then be plotted and processed further for subsequent analysis. Thus, unlike analytical solutions, numerical solutions do not yield compact formulas. Nonetheless, numerical methods are able to handle a much larger class of ordinary differential equations. We begin the chapter with the problems in which all the fixed conditions are set at the initial point, for example, t = 0 or x = x0 , depending on which the independent variable is. These problems are known as initial value problems, or IVP for short. We discuss some of the better known methods for solving initial value problems, such as the one-step methods (e.g., Euler methods and Runge-Kutta methods) and multistep methods (e.g., the Adams-Bashforth methods and Adams-Moulton methods). The actual development of the different methods came out of several approaches, including rigorous algebraic manipulations and collocation methods. However, we implement a simplistic approach by implementing the general numerical methods to a specific differential equation given by dy = f (y) = y dt to generate the required parameters. The main advantage is that a Taylor series expansion of the solution turns out to be sufficiently differentiable and rich in information to produce several necessary conditions. Surprisingly, applying this technique on various numerical methods yields a majority of the necessary conditions, matching those coming from the more elegant, yet lengthy, approaches. In some cases, it yields the same complete sufficient equations, for example, for the Adam’s multistep and backward difference formula (BDF) methods, including the generation of the coefficients for variable step-size BDFs. 273

274

Numerical Solution of Initial and Boundary Value Problems

We include simple examples to compare the performance of the IVP solvers we discuss. Primarily, we show that even though explicit methods tend to be simpler to apply in several cases, they suffer from stability problems. The implicit versions tend to be more stable, but even some implicit methods may end up being unstable. A section on varying step sizes by error-control strategies is also provided in Section G.5 as an appendix. This includes a brief discussion of embedded formulas in explicit Runge-Kutta schemes. Varying step size is needed to allow the methods to adjust the progression along the simulation to finer steps when accuracy requires it, but the schemes also coarsen the steps in regions where errors can be tolerated by large steps. Next, in Section 7.4, we give a brief discussion on the solution of difference equations and their stability criteria. We use these results and apply them to the standard stability analysis of linear IVP solvers; specifically, we introduce the Dahlquist test. By using this standard, we can compare the stability regions of the different methods, either implicit or explicit and either one-step or multistep. We then turn our attention to showing how we can extend the utility of IVP solvers to solve boundary value problems (BVP). Specifically, we discuss a systematic approach of the shooting method by applying the theory of matrizants for the solution of linear two-point BVPs. It is then a simple matter to include Newton methods to extend the shooting method to nonlinear two-point BVPs. Finally, we include a brief section on differential-algebraic equations (DAE). We limit the discussion to problems that are classified as index-1, and we outline both a BDF approach and Runge-Kutta approach. In Section G.1, we include a supplementary section of a very brief tutorial on how the initial value problems, boundary value problems, and differential algebraic problems can be solved using the built-in solvers in MATLAB.

7.1 Euler Methods We begin with the simplest approach, called the Euler methods. For the differential equation, dy = f (t, y) dt

(7.1)

subject to the initial condition, y(0) = yo . We can replace the derivative by its finite difference approximation given by  dy 

k y yk+1 − yk ≈ =  dt t=tk

k t tk+1 − tk Let hk = k t = tk+1 − tk for k ≥ 0, then yk+1 = yk + hk f (tk , yk )

(7.2)

Starting with y0 = y(0), then y1 , y2 , . . . can be obtained recursively by implementing (7.2). For simplicity, we now assume a fixed step size, that is, h = hk is constant. However, we do allow the step size to vary when we need to improve computational efficiency using procedures that are collectively known as error control techniques. These topics are included instead in Section G.5 as an appendix. Based on the

7.1 Euler Methods

275

definition of derivatives, accuracy should improve with smaller values of h. However, roundoff errors also impose their own limits on how small h can be, and smaller values of h will require a larger number of computations. Equation (7.2) is known as the explicit Euler method or forward Euler method. A variation to the explicit Euler method can be obtained by using a different approximation of the derivative, yk − yk−1 dy ≈ dt h

→

yk − yk−1 = hf (tk , yk )

or yk+1 = yk + hf (tk+1 , yk+1 )

(7.3)

Equation (7.3) is also known as the implicit Euler Method or backward Euler Method. It is “implicit” because yk+1 appears on both sides of equation (7.3), thus requiring additional steps for the evaluation of yk+1 . Recall from Section 6.5 that a system of higher order differential equation can be recast in a state-space formulation given by d y = f (t, y) dt

(7.4)

where y is the state vector and f is a vector of multivariable functions f i (t, y). The Euler methods are then given by yk+1

=

yk + hf (tk , yk )

: Explicit Euler

(7.5)

yk+1

=

yk + hf (tk+1 , yk+1 )

: Implicit Euler

(7.6)

For the linear case, f(t, y) = A(t)y + b(t), which results in   I + hA(tk ) yk + hb(tk ) yk+1 = yk+1

=

EXAMPLE 7.1.

 −1   yk + hb(tk+1 ) I − hA(tk+1 )

: Explicit Euler : Implicit Euler

(7.7) (7.8)

Consider the simple first-order linear differential equation

dy + y = −e−βt dt subject to y(0) = y0 . The analytical solution is given by   1 1 y(t) = y0 + e−t/τ + e−βτ 1 − βτ βτ − 1 τ

(ExEu)

Let yk

be the value for y(tk ) using the explicit Euler formula, then  h  (ExEu) (ExEu) (ExEu) yk = yk − yk+1 + e−βtk τ   h (ExEu) h −βtk yk = 1− − e τ τ

(7.9)

(7.10)

276

Numerical Solution of Initial and Boundary Value Problems (ImEu)

Let yk

be the value for y(tk ) using the implicit Euler formula, then  h  (ImEu) (ImEu) (ImEu) = yk − yk+1 yk+1 + e−βtk+1 τ     τ h (ImEu) = yk − e−βtk+1 τ+h τ+h

(7.11)

As a particular example, we can set τ = 0.001, β = 100, and y0 = 1.0. Then Figure 7.1 shows the performance of both the explicit and the implicit Euler method for h = 0.0001, 0.001, 0.002. At the critical value of h = τ = 0.001, we note that both explicit and implicit Euler methods are still stable methods, containing larger error values near the initial point. When h = 0.002, the explicit method became unstable, whereas the implicit method was stable. Conversely, for h = 0.0001, the error dropped significantly as expected, but at the cost of increasing the number of computations and amount of storage.

Generally, the explicit methods are used more often than the implicit methods because they avoid the additional steps of solving (possibly nonlinear) equations for yk+1 . Example 7.1 shows that as long as the increments h are sufficiently small, the explicit Euler should be reasonably satisfactory. However, very small values of h imply larger computational loads. The implicit Euler method for Example 7.1 involved only a simple inverse. However, in general, the implicit solution for yk+1 could be more difficult, often involving nonlinear solvers. Nonetheless, as shown in Example 7.1, the implicit methods are more stable. The issues of stability is discussed in more detail in Section 7.4.

7.2 Runge Kutta Methods A class of methods known as the Runge-Kutta methods can improve the accuracy of Euler methods even under the same values of increment h. The main idea is to obtain a linear combination among several evaluations of the derivative functions f(t, y) at intermediate points between tk and tk+1 to obtain a transition (or update) term δk , such as yk+1 = yk + δk As with Euler methods, there are explicit Runge-Kutta methods and implicit RungeKutta methods. Let s ≥ 1 be an integer that we refer to as the number of stages.1 Then the s-stage Runge Kutta method for dy/dt = f (t, y) is given by: 8  9 s  bj δk δkj = hf (tk + a j h) , yk + j = 1, . . . , s (7.12) =1

yk+1

=

yk +

s 

c j δkj

(7.13)

j =1 1

The stage s determines the number of parameters. However, the “order” refers to the accuracy based on the terms used in the Taylor series expansion.

7.2 Runge Kutta Methods

277

h = 0.0001

h = 0.0001

2

0.05

Analytical Solution Explicit Euler Implicit Euler

1

Explicit Euler Implicit Euler

Error

0

y

0

−1

−2

−3 0

0.002

0.004

0.006

0.008

−0.05 0

0.01

0.002

0.004

t

0.006

0.008

0.01

t

h = 0.001

h = 0.001

2

0.8

Analytical Solution Explicit Euler Implicit Euler

1

0.6

Explicit Euler Implicit Euler

0.4

0.2

y

Error

0

−1

0

−0.2

−0.4

−2 −0.6

−3 0

0.002

0.004

0.006

0.008

0.01

−0.8 0

0.002

0.004

t

0.006

0.008

0.01

t

h = 0.002

h = 0.002

2

3

2

1

Analytical Solution Explicit Euler Implicit Euler

Error

0

y

Explicit Euler Implicit Euler

1

0

−1 −1

−2

−3 0

−2

0.002

0.004

0.006

t

0.008

0.01

−3 0

0.002

0.004

0.006

0.008

t

Figure 7.1. Performance comparison of the explicit and implicit Euler methods for example 7.1. The plots on the right column shows the errors of the Euler methods at different values of h.

0.01

278

Numerical Solution of Initial and Boundary Value Problems

where δkj are the intermediate update terms. Thus in (7.13), the update δk is a linear combination of δkj weighted by c j . The parameters of the Runge-Kutta methods are a1 , . . . , as , c1 , . . . , cs , and b11 , b12 . . . , bss . The parameters a j affect only tk , whereas the parameters bj, affect only the yk during the evaluations of f (t, y). All three sets of parameters are usually arranged in a table called the Runge-Kutta Tableau as follows: ⎛ ⎞ a1 b11 · · · b1s ⎜ a ⎟ . .. .. .. B ⎟ ⎜ . . . ⎜ ⎟ = .. (7.14) ⎜ ⎟ a b · · · b ⎝ ⎠ s s1 ss c1 · · · cs cT Based on the structure of matrix B, the type of Runge-Kutta method can be classified as follows: 1. Explicit Runge-Kutta (ERK). If matrix B is strictly triangular, that is, bij = 0, i = 1, . . . , s, j ≥ i, the method is known an explicit Runge-Kutta method. For these methods, the intermediate updates, δkj , given in (7.12) can be evaluated sequentially. 2. Implicit Runge-Kutta (IRK). If matrix B is not strictly triangular, the method is known as an implicit Runge-Kutta (IRK) method. Some of the special cases are: (a) Diagonally Implicit Runge-Kutta (DIRK). bij

=

0

j >i

b

=

0

for some 1 ≤  ≤ s

(7.15)

(b) Singly Diagonally Implicit Runge-Kutta (SDIRK). This is a special case of DIRK in which all the diagonal elements are equal, that is, b = γ,  = 1, . . . , s. Note that a system of nonautonomous equations, that is, dy/dt = f(t, y), can be replaced by an autonomous system dx/dt = g(x) after appending the original state y with t, and extending f by the constant 1, that is,  x=

y



 and

t

g=

f(x) 1

 (7.16)

The advantage of (7.16) is that we will no longer need the parameters a j during the Runge-Kutta calculations. Nonetheless, in some cases, there are still advantages to using the original nonautonomous system because the stability solution properties apply only to y (note that t is allowed to be unbounded). If we require (7.12) to hold for the special case of [y = t, f = 1] and the case [ f (y) = f (t)], we have the following consistency condition: aj =

s  =1

bj

; j = 1, . . . , s

(7.17)

7.2 Runge Kutta Methods

279

The parameters of the Runge-Kutta methods can be obtained by using Taylor series expansisons. The function y(tk+1 ) can be expanded as a Taylor series around y(tk ),    dy  h2 d2 y  h3 d3 y  y(tk+1 ) = y(tk ) + h + + + ··· dt t=tk 2! dt2 t=tk 3! dt3 t=tk In terms of f (t, y) = dy/dt, we have yk+1

h2 = yk + hf (tk , yk ) + 2!



   ∂f  ∂f  f  + + ··· ∂y tk ,yk ∂t tk ,yk

(7.18)

To obtain an n th -order approximation, the series will be truncated after the (n + 1)th term. Thus the Euler method is nothing but a first-order Taylor series approximation. In the next two sections, we discuss the fourth-order explicit Runge-Kutta method and the fourth-order implicit Runge-Kutta method.

7.2.1 Fourth-Order Explicit Runge-Kutta Method The fourth-order explicit Runge-Kutta method is one of the most used form of Runge-Kutta method. Specifically, we refer to the method given by the tableau in (7.19). The details on how these parameters were obtained are given in Section G.2.1 as an appendix. ⎛

⎞

⎜ ⎜ ⎜ ⎜ a ⎜ ⎜ ⎜ ⎜ ⎜ ⎝

⎟ ⎟ ⎟ ⎟ ⎟ ⎟= ⎟ ⎟ ⎟ ⎠

B

0

0

0

0

0

1 2

1 2

0

0

0

1 2

0

1 2

0

0

1

0

0

1

0

1 6

1 3

1 3

1 6

cT

(7.19)

or, written explicitly, δk1

=

δk2

=

δk3

=

δk4

=

hf (tk , yk )   h δk1 hf tk + , yk + 2 2   h δk2 hf tk + , yk + 2 2   hf tk + h, yk + δk3

yk+1

=

yk +

 1 δk1 + 2δk2 + 2δk3 + δk4 6

For the multivariable case d y = f (t, y) dt

(7.20)

280

Numerical Solution of Initial and Boundary Value Problems

where ⎛ ⎜ y=⎝

[1]

⎞

.. .

⎟ ⎠

y

y[M]

⎛   ⎜ ⎜ f t, y = ⎜ ⎝

  f [1] t, y .. .  f [M] t, y

⎞ ⎟ ⎟ ⎟ ⎠

the Runge-Kutta can then be directly generalized as follows:   δk,1 = hf tk , yk   h 1 δk,2 = hf tk + , yk + δk,1 2 2   h 1 δk,3 = hf tk + , yk + δk2 2 2   δk,4 = hf tk + h, yk + δk3  1 yk+1 = yk + δk1 + 2δk2 + 2δk3 + δk4 6

(7.21)

7.2.2 A Fourth-Order Implicit Runge-Kutta Method The Gauss-Legendre method is one type of implicit Runge-Kutta method that has good stability properties. The details for the derivation of the Runge-Kutta parameters are included in Section G.2.2 as an appendix. The Runge-Kutta tableau is given by ⎛ ⎞   √  √  1 3 1 1 3 ⎜ ⎟ − − ⎜ ⎟ 2 6 4 4 6 ⎜ ⎟ ⎜ a ⎟ B ⎜ ⎟  √   √  ⎜ ⎟ ⎜ ⎟ 1 3 1 3 1 ⎜ ⎟= + + (7.22) ⎜ ⎟ 2 6 4 6 4 ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ 1 1 T ⎝ ⎠ c 2 2 or, in actual equations,

δk1

=

⎛

⎞

  √  √  ⎟ ⎜ ⎜ ⎟ 1 3 1 1 3 ⎜ hf ⎜t + − h, yk + δk1 + − δk2 ⎟ ⎟ 2 6 4 4 6 ⎝ ⎠ ⎛

⎞

δk2

=

  √  √  ⎜ ⎟ ⎜ 1 1 3 3 1 ⎟ ⎜ + hf ⎜t + + h, yk + δk1 + δk2 ⎟ 2 6 4 6 4 ⎟ ⎝ ⎠

yk+1

=

yk +

1 (δk1 + δk2 ) 2

(7.23)

7.2 Runge Kutta Methods

281

For the multivariable case, we have δk,1

=

δk,2

=

  hf tk + a1 h, yk + b11 δk,1 + b12 δk,2   hf tk + a2 h, yk + b21 δk,1 + b22 δk,2

yk+1

=

yk +

 1 δk1 + δk2 2

(7.24)

where ⎛

√ 1 3 a1 = − 2 6

;

√ 1 3 a2 = + 2 6

⎜ ⎜ B=⎜ ⎜ ⎝

and

√ 1 3 − 4 6

1 4

√ 1 3 + 4 6

1 4

⎞ ⎟ ⎟ ⎟ ⎟ ⎠

Remarks: Two MATLAB functions, rk4.m and glirk.m, are available on the book’s webpage and implement the fixed-step fourth-order explicit Runge-Kutta and the fourth-order implicit (Gauss-Legedre) Runge-Kutta methods, respectively.

EXAMPLE 7.2. Let us now compare the performance of the fourth-order explicit and implicit Runge-Kutta methods applied to the same differential equation given in Example 7.1, that is,

τ

dy + y = −e−βt dt

(ExRK)

subject to y(0) = y0 . Let yk be the value for y(tk ) using the explicit RungeKutta formula given in (7.20), then   (ExRK) + e−βtk δk1 = h yk   δk1 (ExRK) + δk2 = h yk + e−β(tk +(h/2)) 2   δk2 (ExRK) δk3 = h yk + + e−β(tk +(h/2)) 2   (ExRK) δk4 = h yk + δk3 + e−β(tk +h) (ExRK)

yk+1

(ExRK)

=

yk

+

1 (δk1 + 2δk2 + 2δk3 + δk4 ) 6

(ImRK)

Let yk be the value for y(tk ) using the implicit Runge-Kutta based on the Gauss-Legendre formulas given in (7.23); then, after further rearrangements of the equations, (ImRK)

yk+1 where

 q1 = 1 − R

1 1

(ImRK)

= q1 yk

+ q2 e−βtk



 and q2 = −R

(7.25) e−a1 βh e−a2 βh



282

Numerical Solution of Initial and Boundary Value Problems

√ √ 1 3 1 3 with a1 = − , a2 = + and 2 6 2 6 ⎛   τ 1 ⎜ +  ⎜ h 4 ⎜ 1 1 ⎜ R= ⎜  √  ⎜ 2 2 1 3 ⎜ + ⎝ 4 6

⎞−1 √  1 3 ⎟ − ⎟ 4 6 ⎟ ⎟   ⎟ ⎟ τ 1 ⎟ + ⎠ h 4



Using the parameters used in Example 7.1, τ = 0.001, β = 100, and y0 = 1.0. Figure 7.2 shows the performance of both the explicit and the implicit RungeKutta methods for h = 0.001, 0.002, 0.003. Compared with the Euler method, the accuracy has improved significantly, as expected. At h = 0.001, the Runge-Kutta methods have much smaller errors, even compared with Euler methods using an increment that is ten times smaller – at h = 0.0001. The explicit Euler method had become unstable at h = 0.2, whereas the explicit Runge-Kutta did not. The explicit Runge-Kutta did become unstable when h = 0.03. In all the cases, the implicit Runge-Kutta using the Gauss-Legendre formulation had the best performance. It can be shown later in Section 7.4.2 that the Gauss-Legendre implicit Runge-Kutta method will be stable for all h when τ > 0, but the errors will still increase as h increases. This means that even for the implicit Runge-Kutta method, h may need to be varied to control the errors. A variable step method should be able to improve this approach further. From the previous examples, we observe that both accuracy and stability of the solutions are often improved by using a smaller step size h. However, smaller step sizes require more computation time and storage. Moreover, different step sizes are needed at different regions of the solution to attain the desired level of accuracy while being balanced with computational efficiency. The process of continually adjusting the step sizes of the numerical methods to attain balance between accuracy and computational loads at the appropriate regions of the solution is generally known as error control. Two popular variations to the explicit Runge-Kutta methods that include error control are the Fehlberg 4/5 method and the Dormand-Prince 4/5 method. Details for both these approaches are included in section G.5 of the appendix.

7.3 Multistep Methods Both the Euler methods and Runge-Kutta methods are known as one-step methods because they use only the current estimate yk to predict the next value yk+1 . A more general approach is to include additional past values (i.e., yk−1 , yk−2 , etc.). These methods are known as multistep methods. We limit our development first to the scalar autonomous differential equations, that is, dy/dt = f (y). A general formulation of the m-step multistep method becomes yk+1 =

m  i=0

ai yk−i + h

m  j =−1

bj f (yk−j )

(7.26)

7.3 Multistep Methods

283 h = 0.001

−3

h = 0.001

16

1

x 10

14

Analytical Solution Explicit RK Implicit RK

y

10

Error

0.5

Explicit RK Implicit RK

12

0

8

6

4

−0.5

2

0

−1 0

0.002

0.004

0.006

0.008

−2 0

0.01

0.002

0.004

0.006

t

t

h = 0.002

h = 0.002

0.008

0.01

1 0.4

Analytical Solution Explicit RK Implicit RK

y

0.3

Error

0.5

Explicit RK Implicit RK

0

0.2

0.1

−0.5 0

−1 0

0.002

0.004

0.006

0.008

0.01

−0.1 0

0.002

0.004

t

0.008

0.01

0.008

0.01

h = 0.003

h = 0.003

7

6

5

0.006

t

Analytical Solution Explicit RK Implicit RK

6 Explicit RK Implicit RK

5

4

4

Error

y

3

2

3

2

1

1

0

−1 0

0

0.002

0.004

0.006

t

0.008

0.01

−1 0

0.002

0.004

0.006

t

Figure 7.2. Performance comparison of the explicit and implicit Runge-Kutta methods for example 7.2

284

Numerical Solution of Initial and Boundary Value Problems

If b−1 = 0, then the method is known as an explicit multistep method; otherwise it is known as an implicit multistep method. The form given in (7.26) can be generalized further by allowing nonautonomous functions f (tk−j , yk−j ). However, because our specific development will initially be applied only to f (y) = y, it is sufficient to limit our form to that given in (7.26). We address the nonautonomous cases later during the extension to multivariable cases. One of the most used explicit multistep methods is the Adams-Bashforth method, whereas the most used implicit multistep method is the Adams-Moulton method. In both cases, they assume a0 = 1 and a1 = · · · = am = 0. The order is generally not equal to the value of m. (We show later that the method is a fourthorder Adams-Bashforth method for m = 3 and the method is a fourth-order AdamsMoulton method for m = 2). We also include in this section the idea of using Adams-Bashforth as a “predictor,” followed by Adams-Moulton as a “corrector.” This is one type of a class of methods known as predictor-corrector methods. Finally, we also introduce a group of methods known as the backward-difference formulas (BDF) methods. It can be shown to be a generalization of the backward Euler method but achieves a higher order of accuracy. These methods are also implicit multistep methods.

7.3.1 Adams-Bashforth Method For an n th -order Adams-Bashforth method, let m = n − 1, then the parameters for bk can be obtained by solving the following equation: ⎞ ⎞⎛ ⎞ ⎛ ⎛ 1 1 ··· 1 ⎟ ⎜ b0 ⎟ ⎜ ⎟ ⎜ 1 1 ⎟ ⎟⎜ ⎟ ⎜ ⎜ ⎜ ⎟ ⎟⎜ ⎟ ⎜ ⎜ 1 ⎟ ⎟⎜ ⎟ ⎜ ⎜ ⎟ − 2 ··· m ⎟ ⎜ b1 ⎟ ⎜ ⎜ 0 1 ⎟ 2 ⎟⎜ ⎟ ⎜ ⎜ ⎟ (7.27) ⎟⎜ ⎟=⎜ ⎜ ⎟ ⎜ ⎟ ⎟ ⎜ . . . . . . ⎜ ⎟ .. . ⎜ ⎟ ⎟ ⎜ .. .. .. . . . ⎟ . . ⎟⎜ . ⎟ ⎜ ⎜ ⎟ ⎟⎜ ⎟ ⎜ ⎜ m ⎟ ⎜ ⎝ 0 1 2m · · · mm ⎠ ⎝ b ⎠ ⎝ (−1) ⎠ m m+1 Details on how this equation was obtained are given in Section G.3 as an appendix. The matrix multiplying the vector of b coefficients is a Vandermonde matrix, which is nonsingular. This means that the coefficients will be unique. For the fourth-order Adams-Bashforth method, ⎞⎛ ⎛ ⎞ ⎛ ⎞ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎝

1

1

1

0

1

2

0

1

4

0

1

8

1 ⎟⎜ ⎟⎜ ⎟⎜ ⎟⎜ ⎟⎜ ⎜ 3 ⎟ ⎟⎜ ⎟⎜ ⎟⎜ ⎟⎜ ⎟⎜ 9 ⎟⎜ ⎟⎜ ⎟⎜ ⎟⎜ ⎟⎜ 27 ⎠ ⎝

b0 ⎟ ⎜ 1 ⎟ ⎟ ⎜ ⎟ ⎟ ⎜ ⎟ ⎟ ⎜ ⎟ ⎟ ⎜ 1 ⎟ ⎜ ⎟ ⎟ b1 ⎟ ⎜ − 2 ⎟ ⎟ ⎜ ⎟ ⎟=⎜ ⎟ ⎟ ⎜ ⎟ 1 ⎟ ⎟ ⎜ ⎟ b2 ⎟ ⎜ ⎟ ⎜ 3 ⎟ ⎟ ⎜ ⎟ ⎟ ⎜ ⎟ ⎟ ⎜ ⎟ 1 ⎠ b3 ⎠ ⎝ − 4

7.3 Multistep Methods

285

which yields 55 −59 37 −9 b1 = b2 = b3 = 24 24 24 24 Thus the fourth-order Adams-Bashforth method is given by   h yk+1 = yk + 55f (yk ) − 59f (yk−1 ) + 37f (yk−2 ) − 9f (yk−3 ) 24 b0 =

or for the multivariable case,   h yk+1 = yk + 55 f(yk ) − 59 f(yk−1 ) + 37 f(yk−2 ) − 9 f(yk−3 ) 24

(7.28)

(7.29)

Based on the multivariable case, we could handle the nonautonomous case by including t as one of the elements in the state vector y, that is, with y[M] = t and dy[M] /dt = 1. It turns out, however, that with f [M] = 1, (7.28) simply reduces to [M]

[M]

yk+1 = yk

+h

which is just the same equation as tk+1 = tk + h. This means that the AdamsBashforth formulas can be extended immediately to handle the nonautonomous case, that is,   h 55 f(tk , yk ) − 59 f(tk−1 , yk−1 ) + 37 f(tk−2 , yk−2 ) − 9 f(tk−3 , yk−3 ) yk+1 = yk + 24 (7.30)

7.3.2 Adams-Moulton Method For the Adams-Moulton method, the procedure is similar except for the inclusion of b−1 . After this modification, the resulting matrix equation is given by ⎛ ⎞⎛ ⎞ ⎛ ⎞ 1 1 1 1 ··· 1 ⎜ ⎟ ⎜ b−1 ⎟ ⎜ ⎟ ⎜ ⎟⎜ ⎟ ⎜ ⎟ ⎜ ⎟⎜ ⎟ ⎜ ⎟ 1 ⎜ ⎟⎜ ⎟ ⎜ ⎟ − −1 0 1 ··· m ⎟ ⎜ b0 ⎟ ⎜ ⎜ ⎟ 2 ⎜ ⎟⎜ ⎟ ⎜ ⎟ ⎜ ⎟⎜ ⎟=⎜ ⎟ (7.31) ⎜ ⎜ ⎜ ⎟ ⎟ ⎟ .. .. .. .. .. .. ⎟ ⎜ .. ⎜ ⎜ ⎟ ⎟ . . . . . . ⎜ ⎟⎜ . ⎟ ⎜ ⎟ ⎜ ⎟⎜ ⎟ ⎜ (−1)m+1 ⎟ ⎝ (−1)m+1 ⎠ 0 1 · · · mm+1 ⎠ ⎝ bm ⎠ ⎝ m+2 For the second-order Adams-Moulton method, we have ⎞⎛ ⎛ ⎞ ⎛ ⎞ b−1 1 1 1 ⎟ ⎟⎜ ⎜ ⎟ ⎜ ⎟⎜ ⎜ ⎟ ⎜ ⎟ −→ b−1 = b0 = 1 ⎟⎜ ⎜ ⎟= ⎝ − 1 0 ⎠⎝ b ⎠ ⎝ 2 1 ⎠ 0 − 2 or yk+1

h = yk + 2

     f yk+1 + f yk

This case is also known as the trapezoidal method.

(7.32)

286

Numerical Solution of Initial and Boundary Value Problems

For the fourth-order Adams-Moulton method, ⎞⎛

⎛ 1

⎜ ⎜ ⎜ ⎜ −1 ⎜ ⎜ ⎜ ⎜ ⎜ 1 ⎜ ⎜ ⎜ ⎝ −1

1

1

0

1

0

1

0

1

1

⎞ b−1

⎟⎜ ⎟⎜ ⎟⎜ ⎜ 2 ⎟ ⎟ ⎜ b0 ⎟⎜ ⎟⎜ ⎟⎜ ⎜ 4 ⎟ ⎟ ⎜ b1 ⎟⎜ ⎟⎜ ⎠⎝ 8 b2

⎛

⎞

1

⎟ ⎜ ⎟ ⎟ ⎜ ⎟ ⎟ ⎜ ⎟ ⎟ ⎜ −1 ⎟ ⎟ ⎜ ⎟ 2 ⎟ ⎟ ⎜ ⎟=⎜ ⎟ ⎟ ⎜ ⎟ ⎟ ⎜ +1 ⎟ ⎟ ⎜ 3 ⎟ ⎟ ⎜ ⎟ ⎟ ⎜ ⎟ 1 ⎠ ⎠ ⎝ − 4

which yields b−1 = 9/24, b0 = 19/24, b1 = −5/24, and b2 = 1/24. Thus the fourthorder Adams-Moulton method is given by yk+1

h = yk + 24



 9f yk+1



      + 19f yk − 5f yk−1 + f yk−2

 (7.33)

Note that because multistep methods require past values of y, one still needs to use one-step methods such as the Runge-Kutta methods for the initialization step. This means that for a fourth-order Adams-Bashforth or Adams-Moulton method, we need to use one-step methods to evaluate y1 , y2 , and y3 (in addition to y0 ) before the Adams-Bashforth could proceed. For the fourth-order Adams-Moulton method, we need the initial values of y0 , y1 , and y2 . Notice that with y = t and f (y) = 1, we also end up with tk+1 = tk + h. Thus the formulas can be extended immediately to include nonautonomous cases and multivariable cases as follows:           h yk+1 = yk + 9 f tk+1 , yk+1 + 19 f tk , yk − 5 f tk−1 , yk−1 + f tk−2 , yk−2 24 (7.34)

7.3.3 Adams Predictor-Corrector Methods Explicit formulas often have the advantage of direct evaluations, whereas implicit formulas usually have the advantages of improved stability. To merge the two methods, one could first use explicit formulas as predictors and then use the implicit formulas as correctors. This is done by applying the predicted value on one side of the implicit formulas, yielding a new value of the updates before moving to the next integration step. To be specific, we can apply the Adams formulas as follows: 1. Predictor: zk+1

=

     h 55 f tk , yk − 59 f tk−1 , yk−1 yk + 24     + 37 f tk−2 , yk−2 − 9 f tk−3 , yk−3

(7.35)

7.3 Multistep Methods

2. Corrector: =

yk+1

287

     h 9 f tk+1 , zk+1 + 19 f tk , yk 24     − 5 f tk−1 , yk−1 + f tk−2 , yk−2

yk +

(7.36)

One could vary the correction stage by performing additional iterations, for example, (0) with wk+1 = zk+1 , (j +1)

wk+1 = yk + (j +1)

      h   (j )  9f wk+1 + 19f yk − 5f yk−1 + f yk−2 24

(7.37) (j +1)

(j )

until wk+1 − wk+1 < , where  is a chosen tolerance, and then set yk+1 = wk+1 . (j +1)

Even though this may converge to the stable value of wk+1 , it may still not necessarily be an accurate value. Instead of (7.37), it is more efficient to decrease the step-size h, using error-control strategies such as those discussed in Section G.5.

7.3.4 Backward-Difference Formula (BDF) Methods Another important set of implicit multistep methods is the backward-difference formula (BDF) method (also known as the Gear’s method). Based on the general form of multistep methods given in (7.26), yk+1 =

m 

ai yk−i + h

m 

  bj f yk−j

j =−1

i=0

the BDF method sets bj = 0 for j ≥ 0, reducing the general formula to be yk+1 =

m 

  ai yk−i + hb−1 f yk+1

(7.38)

i=0

As shown next, we need m = p − 1 to obtain p th -order accuracy. Following the same approach as before, we choose the simplest function for f to be f (t, y) = y. Doing so, we get the analytical solution yk+ = eh yk Substituting these values of yk− yk+1

=

m 

yk− = e−(+1)h yk+1   into (7.38), with f yk+1 = yk+1 , we have →

αi e−(i+1)h yk+1 + hb−1 yk+1

i=0

1

=

hb−1 +

m 

⎛ αi ⎝1 +

i=0

∞  [− (i + 1) h]q

q!

q=1

⎞ ⎠

To obtain p th -order accuracy, we can let m = p − 1 and truncate terms with h p +1 and higher. This yields p −1  i=0

αi = 1 ;

p −1  i=0

αi [− (i + 1)] + b−1 = 0 ;

p −1  [− (i + 1)]q i=0

q!

αi = 0 , q = 2, . . . , p

288

Numerical Solution of Initial and Boundary Value Problems

These equations can be put in matrix form, as given here: ⎞⎛ ⎞ ⎛ ⎛ 0 1 1 ... 1 b−1 ⎟ ⎜ ⎜ −1 1 2 ⎜ ... p ⎟ ⎟ ⎜ a0 ⎟ ⎜ ⎜ ⎜ 0 1 22 . . . p 2 ⎟ ⎜ a1 ⎟ ⎜ ⎟⎜ ⎟=⎜ ⎜ ⎟⎜ . ⎟ ⎜ ⎜ .. .. .. .. .. . ⎠ ⎝ ⎝ . . . . . ⎠ ⎝ . 0

1

2p

...

pp

a p −1

1 0 0 .. .

⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠

(7.39)

0

For a fourth-order BDF, ⎛

⎛ ⎜ ⎜ ⎜ ⎜ ⎝

b−1 a0 a1 a2 a3

⎛

⎞

⎟ ⎜ ⎟ ⎜ ⎟=⎜ ⎟ ⎜ ⎠ ⎝

or yk+1 =

0 −1 0 0 0

1 1 1 1 1

1 2 22 23 24

1 3 32 33 34

1 4 42 43 44

⎞−1 ⎛ ⎟ ⎟ ⎟ ⎟ ⎠

1 ⎜ 0 ⎜ ⎜ 0 ⎜ ⎝ 0 0

12 ⎞ ⎜ 25 ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ 48 ⎜ ⎟ ⎞ ⎜ 25 ⎟ ⎜ ⎟ ⎟ ⎟ ⎜ ⎜ ⎟ ⎟ ⎜ 36 ⎟ ⎟=⎜ − ⎟ ⎟ ⎜ ⎟ 25 ⎠ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ 16 ⎜ ⎟ ⎜ ⎟ 25 ⎜ ⎟ ⎜ ⎟ ⎝ 3 ⎠ − 25

 48 36 16 3 12  yk − yk−1 + yk−2 − yk−3 + hf yk+1 25 25 25 25 25

(7.40)

The mulitvariable case is given by yk+1 =

 48 36 16 3 12  yk − yk−1 + yk−2 − yk−3 + h f yk+1 25 25 25 25 25

(7.41)

BDF methods can handle several stiff differential equations well, but stability properties do deteriorate at higher orders; that is, they are only stable for orders less than 7. Unfortunately, care is needed when extending this result to nonautonomous differential equations because setting y = t and f (y) = 1 will now yield tk+1 =

48 36 16 3 12 tk − tk−1 + tk−2 − tk−3 + h 25 25 25 25 25

which does reduce to tk+1 = tk + h only if the step size h is held constant. Thus for fixed h, the nonautonomous approach implies the simple replacement of f(yk+1 ) by f (tk+1 , yk+1 ). When the step sizes are variable, the coefficient will also have to vary at each step. The generalized formulas for the BDF coefficients when the step sizes vary are discussed in Section G.4. EXAMPLE 7.3. We now compare the performance of the four multistep methods just discussed, applied to the same linear scalar system used in Examples 7.1 and 7.2,

1 dy + y = −e−βt τ dt

7.3 Multistep Methods

We have the following iteration equations: 1. Adams-Bashforth (AB)    h   yk+1 = yk + 55 −yk − e−βtk − 59 −yk−1 − e−βtk−1 24τ     + 37 −yk−2 − e−βtk−2 − 9 −yk−3 − e−βtk−3 2. Adams-Moulton (AM)    24τ h   −βtk+1  yk+1 = yk + 9 −e + 19 −yk − e−βtk 24τ + 9h 24τ     − 5 −yk−1 − e−βtk−1 + −yk−2 − e−βtk−2 3. Adams Predictor-Corrector (APC)    h   zk+1 = yk + 55 −yk − e−βtk − 59 −yk−1 − e−βtk−1 24τ     + 37 −yk−2 − e−βtk−2 − 9 −yk−3 − e−βtk−3    h   yk+1 = yk + 9 −zk+1 − e−βtk+1 + 19 −yk − e−βtk 24τ     − 5 −yk−1 − e−βtk−1 + −yk−2 − e−βtk−2 4. Backward-Difference Formula (BDF)   τ h −βtk+1 yk+1 = 48yk − 36yk−1 + 16yk−2 − 12 e 25τ + 12h τ Figure 7.3 shows the performance of these iteration formulas of the various multistep methods for τ = 0.001, β = 100, and y0 = 1.0. In practical use, one would use one-step methods such as the Runge-Kutta methods to evaluate the first four or three iterations. For our example here, however, we used the exact values coming from the analytical solution to allow us to assess the performance of the multistep methods, independent of the Runge-Kutta methods. All the methods performed well for h = 0.0001. However, when compared with Figure 7.3 for the Runge-Kutta methods, the accuracy and stability of the explicit Adams-Bashforth method did not appear to be as good at h = 0.001. The stability of the predictor-corrector formulas is better than that of the explicit Adams-Bashforth method, but it was unstable when h = 0.002. The two implicit methods are the Adams-Moulton and BDF methods. It appears for small h values, the Adams-Moulton performs better than the BDF method. However, as we see in Figure 7.4, the BDF maintained stability even at h = 0.004, but the Adams-Moulton did not. This shows that although stability is improved by using implicit methods, some methods have greater stability ranges than others. This is one reason that, among multistep methods, the BDF methods are often chosen to handle stiff2 differential equations, and then they are coupled with other enhancements, such as step-size control, to improve accuracy.

2

See Section 7.4 for a description of stiff differential equations.

289

290

Numerical Solution of Initial and Boundary Value Problems h = 0.0001 2.5

Analytical Adams−Bashforth Adams−Moulton Predictor−Corrector BDF

1.5

1

Error

y

x 10

2

Analytical Solution Adams−Bashforth Adams−Moulton Predictor−Corrector BDF

0.5

h = 0.0001

−5

1

0

0.5

0

−0.5

−0.5

−1

−1 0

0.002

0.004

0.006

0.008

−1.5 0

0.01

0.002

0.004

0.006

t h = 0.001

0.01

h = 0.001

1

5 Analytical Adams−Bashforth Adams−Moulton Predictor−Corrector BDF

Analytical Solution Adams−Bashforth Adams−Moulton Predictor−Corrector BDF

Error

0.5

y

0.008

t

0

0

−0.5

−1 0

0.002

0.004

0.006

0.008

−5 0

0.01

0.002

0.004

t h = 0.002

0.008

0.01

0.008

0.01

h = 0.002

1

1

Analytical Solution Adams−Bashforth Adams−Moulton Predictor−Corrector BDF

0.5

0

−0.5

Error

0.5

y

0.006

t

0

−1

−1.5

−2

−0.5

−2.5

Analytical Adams−Bashforth Adams−Moulton Predictor−Corrector BDF

−3

−1 0

0.002

0.004

0.006

t

0.008

0.01

−3.5 0

0.002

0.004

0.006

t

Figure 7.3. Performance comparison of the various multistep methods for example 7.3

7.4 Difference Equations and Stability h = 0.004

h = 0.004

1

0.5

291

0.6

Analytical Solution Adams−Moulton BDF

0.4

Analytical Solution Adams−Moulton BDF

y

Error

0.2

0

0

−0.2

−0.5 −0.4

−1 0

0.05

0.1

0.15

t

0

0.05

0.1

t

Figure 7.4. Performance comparison between Adams-Moulton and BDF at h = 0.04 for example 7.3

7.4 Difference Equations and Stability The numerical methods presented in the previous sections are approximations of the exact solutions in the form of iteration schemes. These iteration equations are actually difference equations. The stability properties of the various approaches will depend on the properties of difference equations rather than on properties of differential equations. In the sections that follow, we start with a brief overview of difference equations, with particular attention to their stability properties. Once the general stability results have been established, we turn to the technique known as the Dahlquist test which involves the testing of particular numerical solution techniques to one of the simplest differential equation dy = λy dt One can then build the stability regions in the complex plane. When looking at the stability regions, one could argue that once we have an accurate method such as the fourth-order Runge Kutta method, we should just be able to vary step size h to maintain stability. The implication of this idea is very problematic for some differential equations, because the step size needed to maintain stability for the chosen method could mean a very small step size. This is true even when error-control strategies are incorporated to numerical methods such as the 4/5th embedded Runge-Kutta methods. The result is what appears to be a “hanging" process because the schemes might have entered a region requiring very small values of h. These systems are generally described as “stiff differential equations.” The general strategy that is often prescribed is to first use explicit high-order methods because these methods are quick and accurate, and there are still a large class of nonstiff systems in several applications. However, when the appearance of “hanging” process occurs, then the system can be deemed “stiff,” and implicit methods are used instead.

0.15

292

Numerical Solution of Initial and Boundary Value Problems

7.4.1 Linear Difference Equations with Constant Coefficients We begin with the definition of the shift operator, Q,   Q yn = yn+1

(7.42)

where yn = y(nh) and h = t. A p th -order linear difference equation with constant coefficients has the following form  p   α p Q p yn = f (n) (7.43) i=0

where αi , i = 0, . . . , p are constant. If f (n) = 0, then we say that (7.43) is homogeneous. Using the constants given in the left-hand side of (7.43), we can define the characteristic equation as p 

αp σ p = 0

(7.44)

i=0

The roots of (7.44) will be used to form the complementary solution, that is, the solution of the homogeneous part of (7.43), as stated by the following theorem: Let f (n) = 0 in the difference equation (7.43), and let the distinct (possibly repeated) roots of the p th -order characteristic equation in (7.44) be σ = ; :  σ1 , . . . , σM where σ j is a k j -fold root (i.e., repeated (k j − 1) times and M j =1 k j = p ). The solution of the homogeneous difference equation is then given by

THEOREM 7.1.

yn = Qn y0 =

M 

S (j, n)

(7.45)

j =1

where

⎛ S (j, n) = ⎝

k j −1



⎞ c j, n  ⎠ (σ j )n

(7.46)

=0

and c j, are arbitrary constants that are used to fit initial conditions. PROOF.

(See Section G.6 for proof.)

EXAMPLE 7.4.

Consider the following difference equation: yn+5 − 2.5yn+4 + 1.96yn+3 − 7.6 × 10−2 yn+2 − 5.453 × 10−1 yn+1 + 1.7885 × 10−1 yn = 0

subject to the following initial conditions: y−4 = y−3 = y−2 = 1, y−1 = y0 = 0. The roots of the characteristic equations are: −0.5, 0.7, 0.7, 0.8 + 0.3i, 0.8 − 0.3i. Following the solution provided by Theorem 7.1, we have yn = C1 (−0.5)n + (C2 + C3 n) (0.7)n + C4 (0.8 + 0.3i)n + C5 (0.8 − 0.3i)n

7.4 Difference Equations and Stability

293

5

4

3

y

n

Figure 7.5. Plot of points from difference solution (open circles) and data obtained by recursion of the difference equation (solid line).

2

1

0

−1

0

10

20

30

n

or using the polar form and Euler’s identity on the complex roots, yn = C1 (−0.5)n + (C2 + C3 n) (0.7)n + rn (A cos (nθ) + B sin (nθ)) where

! r = 0.82 + 0.32

;

−1



θ = tan

0.3 0.8



To match the initial conditions, we obtain: C1 = 4.9052 × 10−2 , C2 = 1.0209 × 101 , C3 = 2.4539, A = −1.0258 × 101 , and B = −6.3538 × 10−1 . A plot of the solution is given in Figure 7.5, together with the curve obtained by iteration of the difference equation. Recall that the asymptotic stability of differential equations requires that the real part of all eigenvalues are negative. This is not the case for difference equations. The stability of difference equations instead requires the roots of the characteristic equation to have magnitudes that are less than unity.3 In this example, four of the eigenvalues have positive real parts, but all the roots have magnitudes less than 1. Thus the process is stable.

Let DN be the determinant of matrix AN , where AN is the N × N tri-diagonal matrix given by ⎞ ⎛ a b 0 ⎟ ⎜ ⎟ ⎜ c ... ... ⎟ AN = ⎜ ⎟ ⎜ .. .. ⎝ . . b ⎠

EXAMPLE 7.5.

0

c

a

By expanding along the first column, we can show that DN = aDN−1 − bcDN−2 2

See Theorem 7.2.

or

DN+2 − aDN+1 + bcDN = 0

40

294

Numerical Solution of Initial and Boundary Value Problems

which is a difference equation for N ≥ 1, with D1 = a and D2 = a2 − bc. The characteristic equation is σ2 − aσ + bc = 0 whose two roots can be a complex-conjugate pair, a set of repeated realnumbers, or a pair of different real numbers, depending on whether the term a2 − 4bc is negative, zero, or positive, respectively. Take √ the specific case of a = 3, b = −2, c = −2. Then the roots are σ = 1.5 ± i 7/2 yielding   3 N DN = 2 cos (Nθ) + √ sin (Nθ) 7 √  7 where θ = tan−1 . 3 The particular solutions of difference equations are usually found by the method of undetermined coefficients. Suppose the nonhomogeneous function of (7.43) is given by f (n) = a nq bn cos (nθ)

or

f (n) = a nq bn sin (nθ)

(7.47)

where a, b, q, and θ are constants, with q as a positive integer. The particular solution, Y , can then be formulated as     (7.48) Y = n K bn (A0 + · · · + Aq nq ) cos (nθ) + (B0 + · · · + Bq nq ) sin (nθ)   where K = 0 if R = beiθ is not a root of the characteristic equation (7.44). If R = beiθ = σ j , where σ j is a k j -fold root of the characteristic equation (7.44), then we need to set K = k j . The coefficients A0 , . . . , Aq and B0 , . . . , Bq can then be determined after substituting Y into the difference equation.

7.4.2 Stability As long as the term b in (7.47) has a magnitude less than 1, f (n) is bounded as n → ∞. This means that the particular solution will also be bounded as n → ∞. One can then conclude that as long as the nonhomogeneous term f (n) is bounded, the source of instability can only come from the complementary solution. Using Theorem 7.1, the stability of a linear difference equation will then depend on the roots of the characteristic equation. For the linear difference equation (7.43), with f (n) as a linear combination of terms having the form given in (7.47), let f (n) < ∞ as n → ∞, and let σ = (σ1 , . . . , σM ) be the set of distinct roots of the characteristic equation (7.44). Then the solution of the difference equation, yn , is stable if 4    σj  < 1 for j = 1, . . . , M THEOREM 7.2.

4

The theorem gives a sufficient condition. For each σ j that is not repeated, the stability condition could be relaxed to be |σ j | ≤ 1. However, because round-off errors are usually present, it may be more reasonable to use the strict inequality when applying it to the stability analysis of numerical solutions.

7.4 Difference Equations and Stability

295

The extension to linear multivariable difference equations is straightforward. The standard form, however, is to put it back into the familiar state-space formulation, that is, xn+1 = Axn + bn where

(7.49)

⎛

⎞ x1n ⎟ ⎜ xn = ⎝ ... ⎠ xM n ⎛

··· .. . ···

a11 ⎜ .. A=⎝ . aM1

⎞ a1M .. ⎟ . ⎠ aMM

Let matrix T be the nonsingular matrix that would yield the Jordan canonical decomposition of A: TJT −1 = A. The solution of (7.49) is x1

= .. .

Ax0 + b0

xn

=

A x0 + n

n−1 

Ai−1 bi−1

i=0

or xn = TJ n T −1 x0 +

n−1 

TJ i−1 T −1 bi−1

(7.50)

i=1

When J is diagonal, ⎛ σ1 ⎜ .. J =⎝ . 0 +

⎞

0

⎛

⎟ ⎠

σM n−1 

→

σ1n

⎜ xn = T ⎝

0 ..

.

⎜ T⎝

i=1

σ1i

0 ..

.

⎟ −1 ⎠ T x0

n σM

0 ⎛

⎞

⎞ ⎟ −1 ⎠ T bi−1

i σM

0

When J is a full Jordan block (cf. (3.36)), ⎞ ⎛ σ 1 0 ⎟ ⎜ .. .. ⎟ ⎜ . . ⎟ → J = diag ⎜ ⎟ ⎜ .. ⎝ . 1 ⎠ σ

0 ⎛ xn

=

⎜ ⎜ T⎜ ⎜ ⎝

σn

βn,n−1 σn−1 .. .

⎛ +

n−1  i=1

⎜ ⎜ T⎜ ⎜ ⎝

···

βn,n−M+1 σn−M+1

..

βn,n−M+2 σn−M+2 .. .

.

⎞ ⎟ ⎟ −1 ⎟ T x0 ⎟ ⎠

σn σi

βi,i−1 σi−1 .. .

···

βi,i−M+1 σi−M+1

..

βi,i−M+2 σi−M+2 .. .

.

σi

⎞ ⎟ ⎟ −1 ⎟ T bi−1 ⎟ ⎠

296

Numerical Solution of Initial and Boundary Value Problems

where

βk,j =

⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩

k! (k − j )!j ! 0

if j ≥ 0 otherwise

In either case, the stability again depends on the eigenvalues of A; that is, as long as bi in (7.49) are bounded, then stability is guaranteed if |σi | < 1 for all i = 1, . . . , M. Based on the preceding canonical forms, for most numerical methods, including single-step or multistep, implicit or explicit, the stability is applied to the the Dahlquist test case. For the single-step methods, the test involves the following steps: 1. Apply the chosen numerical method on dy = λy dt with y(0) = 1. 2. With h = t and z = λh, rearrange the equation to be yn+1 = g (z) yn 3. Find the stability region, which is the region such that   * ) Stab : z : g (z) < 1 When the s-stage and s-order explicit Runge-Kutta methods are applied to the Dahlquist test case, the difference equation becomes  s  1 i yn+1 = (λh) yn i!

EXAMPLE 7.6.

i=0

The stability region is then shown in Figure 7.6 for s = 1, 2, 3, 4. Note that, although the stability region increases with the number of stages s, the explicit Runge-Kutta methods will be conditionally stable; that is, the step size is constrained by the value of λ. Applying the backward Euler method to the Dahlquist test, the difference equation becomes yn+1 =

1 yn 1 − λh

The stability region is shown in Figure 7.7. This means that if Real (λ) < 0, the backward Euler method will be stable for any step size. Finally, we can assess the stability region for the Gauss-Legendre method, which is an implicit fourth-order Runge-Kutta. The difference equation after applying the Gauss-Legendre method to the Dahlquist test is given by yn+1 =

12 + 6λh + (λh)2 12 − 6λh + (λh)2

yn

7.4 Difference Equations and Stability

297 3 s=4 s=3

2

s=2

Figure 7.6. Stability region for z = λh using the explicit RungeKutta methods (unshaded regions) of order s = 1, 2, 3, 4.

Imag ( z )

1 s=1

0

−1

−2

−3 −3

−2

−1

0

1

Real ( z )

The stability region is shown in Figure 7.8. For Real (λ) < 0, the Gauss-Legendre method will also be stable for any step size h.

For multistep methods, the procedure is similar. Instead, the characteristic equations will yield multiple roots, which are functions of z = λh. One of the roots, usually denoted by σ1 , will have a value of 1 at z = z∗ . This root is known as the principal root. The other roots are known as the spurious roots. For accuracy and convergence properties, the principal root is the most critical, whereas the effects of spurious roots die out eventually, often quickly, as long as the method is stable. The stability regions can again be obtained by applying the numerical method to the Dahlquist test case to obtain the characteristic equation. If any of the roots at z = z∗ , σk (z∗ ), have a magnitude greater than 1, then z∗ belongs to the unstable region.

2

1.5

Figure 7.7. Stability region for z = λh using the backward Euler methods (unshaded regions).

Imag (z)

1

0.5

0

−0.5

−1

−1.5

−2 −1

0

1

Real (z)

2

3

298

Numerical Solution of Initial and Boundary Value Problems 10

Imag (z)

5

Figure 7.8. Stability region for z = λh using the Gauss-Legendre methods (unshaded regions).

0

−5

−10 −10

−5

0

5

10

Real (z)

Let us apply the Dahlquist test case, that is, f (t, y) = λy, to the fourth-order Adams-Moulton method given in (7.33); then we obtain the following difference equation: z yk+1 = yk + (9yk+1 + 19yk − 5yk−1 + yk−2 ) 24 where z = λh, and whose characteristic equation is then given by         9 19 5 1 3 2 −1 + z σ + 1 + z σ − z σ+ z =0 24 24 24 24

EXAMPLE 7.7.

Because the characteristic equation is linear in z, it is simpler to determine the boundary separating the stable region from the unstable region by finding z as a function of σ = eiθ , −π ≤ θ ≤ π, that is,   24 e3θi − e2θi z (θ) = 3θi 9e + 19e2θi − 5eθi + 1 Thus for the Adams-Moulton implicit method, this region is shown in Figure 7.9. Note that, as suggested in Example 7.3, specifically in Figure 7.4, even though the Adams-Moulton is an implicit method, the stability range is still limited, thereby making it still not suitable for solving stiff differential equations.

From the stability regions shown in Example 7.6, we note that the explicit RungeKutta methods will need small values of step size as Real (λ) becomes more negative. However, both the backward Euler and the Gauss-Legendre methods are unconditionally stable for Real (λ) < 0. This type of stability is also known as A-stability. There are other types of stability, some of which are shown in Figure 7.10. Some of these alternative types of stability are needed to discriminate among different schemes, especially multistep methods. As we have noted, for some numerical methods, especially explicit methods, stability requirements may demand smaller time steps than accuracy requires. When this situation occurs, we say that the system is a stiff differential equation. In several cases, differential equations are classified as stiff when the difference between the

7.5 Boundary Value Problems

299

2

1.5

Untable

Figure 7.9. The stabliity region of the AdamsMoulton implicit method for the Dahlquist test case.

Imag (z)

1

0.5

Stable

0

−0.5

−1

−1.5

−2

−3

−2

−1

Real (z)

magnitudes of the largest and smallest eigenvalues is very large. For stiff differential equations, implicit methods such as the Gauss-Legendre IRK or multistep BDF schemes are usually preferred. Specifically, D-stability (cf. Figure 7.10), that is, where the stability is guaranteed inside the region whose real part is less than D < 0, is also known as stiffly stable. It can be shown that BDF schemes of lower order (e.g., order ≤ 6) are stiffly stable. Finally, note that the stability issues discussed in this section are all based on linear systems. For nonlinear systems, linear stability analysis can be applied locally via linearization. Other sophisticated approaches are needed for global analysis and are not covered here.

7.5 Boundary Value Problems Most of the numerical methods discussed thus far are based on the assumption that all the conditions are given at one point, for example, at t = 0. Those problems are known as initial value problems (IVP). The numerical methods could also be applied at t = T and integrated backward by appropriate changes of the independent variables. For instance, with τ = T − t, the problem becomes an initial value problem with respect to τ. However, there are several applications in which some of the conditions are specified at more than one point. For example, some conditions may be set at t = 0, and the rest are set at t = T . These problems are known as boundary value problems (BVP). When this is the case, we cannot just use the initial value problems (IVP) solvers such as Runge-Kutta methods, Adams methods, or BDF methods directly. We limit our discussion to two-point boundary value problems: one at t = 0 and the other at t = T . There are two types of boundary conditions: 1. Separated boundary conditions. q0 (x(0))

=

0

qT (x(T ))

=

0

(7.51)

0

300

Numerical Solution of Initial and Boundary Value Problems

Im( h)

Im( h)

Re( h)

Re( h)

0

-D

0

Im( h)

Im( h)

Re( h)

Re( h)

0

0

0 Figure 7.10. Different types of numerical stability. The unshaded regions are stable; for example, for Aα -stability, the shaded regions do not cross the dotted lines.

where q0 and qT are general nonlinear functions. In the linear case, we have Q0 x(0)

=

β0

QT x(T )

=

βT

(7.52)

where Q0 and QT are constant matrices of sizes (k × n) and [(n − k) × n], respectively. 2. Mixed boundary conditions. q (x(0), x(T )) = 0

(7.53)

where q is a nonlinear function. In the linear case, we have Qa x(0) + Qbx(T ) = p

(7.54)

where Qa and Qb are constant square matrices of size n × n. In the next section, we discuss the shooting method for linear boundary value problems, which we later generalize to handle the nonlinear boundary value problems. Another approach, known as the Ricatti method, is included in Section G.8 as

7.5 Boundary Value Problems

301

an appendix. Yet another approach is the finite difference method, which is discussed in Chapter 13.

7.5.1 Linear Boundary Value Problems Consider the general linear differential equation, d x = A(t)x + b(t) dt

(7.55)

subject to a mixed boundary conditions at t = 0 and t = T , given by Qa x(0) + Qbx(T ) = p

(7.56)

where x is the vector of n state variables and p is a vector of n constants. To build upon the tools available for initial value problems, we can pose the problem as the search for initial conditions, x(0), such that (7.56) will be satisfied. This approach is known as the shooting method. Recall from Section 6.5.3 that the solution of (7.55) is given by x(t) = M(t)x(0) + z(t)

(7.57)

where M(t) is the matrizant of the system that satisfies the homogenous part, that is, d M = A(t)M dt

;

M(0) = I

(7.58)

and z is the particular solution given by  z(t) = M(t)

t

M(τ)−1 b(τ)dτ

;

z(0) = 0

(7.59)

0

Using (7.57), at t = T , x(T ) = M(T )x(0) + z(T )

(7.60)

Substituting (7.60) into the boundary conditions (7.56),   Qa x(0) + Qb M(T )x(0) + z(T ) = p   Qa + QbM(T ) x(0) = p − Qbz(T ) x(0)

=

−1    p − Qbz(T ) Qa + QbM(T ) (7.61)

In case M(T ) and z(T ) can be evaluated analytically, x(0) can be calculated and substituted into (7.57). This would yield an analytical solution for x(t). However, in other cases, one may have to rely on numerical methods to estimate M(T ) and z(T ). To estimate M(T ), apply the properties given in (7.58). Using IVP solvers such as Runge-Kutta methods, Adams methods, or BDF methods, while setting initial conditions to e j (the j th column of the identity matrix), we could integrate the

302

Numerical Solution of Initial and Boundary Value Problems

homogenous part of the differential equation until t = T . This yields m j (T ), that is, the j th column of M(T ). Thus ⎡ ⎞⎤ ⎛ 1 ⎢ ⎜ 0 ⎟⎥  ⎢ d ⎟⎥ ⎜ ⎢ ⎟⎥ ⎜ −→ m1 (t = T ) IVS ⎢ m1 = A(t)m1 , m1 (0) = ⎜ ... ⎟⎥ ⎢ dt ⎟⎥ ⎜ ⎣ ⎝ 0 ⎠⎦ 0 .. .⎛

⎡

0 0 .. .

⎢ ⎜  ⎢ d ⎜ ⎢ ⎜ IVS ⎢ mn = A(t)mn , mn (0) = ⎜ ⎢ dt ⎜ ⎣ ⎝ 0 1

⎞⎤ ⎟⎥ ⎟⎥ ⎟⎥ ⎟⎥ ⎟⎥ ⎠⎦

−→

mn (t = T )

where IVS denotes any initial value solver such as Runge-Kutta methods, Adams methods, or BDF methods. Afterward, we can combine the results to form the matrizant at t = T , that is,   M(T ) =

m1 (T )

m2 (T )

···

mn (T )

Likewise, to estimate z(T ), apply the properties given in (7.59). Using a zero initial condition, we could integrate the nonhomogeneous part of the differential equation until t = T to obtain z(T ), that is, ⎛ ⎡ ⎞⎤ 0   ⎜ ⎟⎥ ⎢ d −→ z (t = T ) IVS ⎣ z = A(t)z + b(t) , x0 = ⎝ ... ⎠⎦ dt 0 Once M(T ) and z(T ) have been estimated, equation (7.61) can be used to determine the required initial condition, x(0). Finally, we can use the initial value solvers once more and integrate the (nonhomogenous) differential equation (7.55) using x(0).

EXAMPLE 7.8.

Consider the following linear differential system of equations:     d 1 1 −5e−3t x + x= 0 −e−t/2 −1 dt

subject to x1 (0) = 0 and x1 (4) − x2 (4) = −4. To put the boundary conditions into the form of (7.56), we have       1 0 0 0 0 ; Qb = ; p= Qa = 0 0 1 −1 −4 Following the procedure, we find   2.9052 × 10−1 4.4240 × 10−2 M(4) = −2.7606 × 10−3 −6.0691 × 10−2

 and

z(4) =

2.4966 × 100 −3.8973 × 10−1



7.6 Differential Algebraic Equations

303

5 x2

0

−5

1

Figure 7.11. Plot of x(t) for the linear boundary value problem of Example 7.8.

x ,x

2

x1

−10

−15

−20

−25 0

1

2

3

t

which can now be applied to solve for the required x(0),  −1    p − Qbz(T ) = x(0) = Qa + QbM(T )

0 20.182



A plot of x(t) is shown in Figure 7.11, which shows that the boundary conditions are indeed satisfied. The extension to nonlinear boundary value problems can be handled by using Newton-type algorithms. Details of these approaches are given in Section G.7.

7.6 Differential Algebraic Equations There are several physical applications that result in a set of differential equations together with a set of nonlinear algebraic equations. For instance, in the field of process control, the dynamics of the system could be described generally by   d fsys t, y, y, u = 0 (7.62) dt where t, y, and u are time, state variable, and control variable, respectively, and fsys and y have the same size. A feedforward control strategy specifies u = u(t) independent of y, and (7.62) is a nonlinear ODE system. Conversely, a feedback strategy could be implemented by specifying a control law given by fctrl (t, y, u) = 0 Taken together, (7.62) and (7.63) become a set of equations of the form ⎛ ⎞   d   ⎜ fsys t, y, dt y, u ⎟ ⎜ ⎟ = f t, z, d z = 0 ⎝ ⎠ dt fctrl (t, y, u)

(7.63)

(7.64)

where z = (y, u)T is an extended state vector. The combined system in (7.64) is an example of a differential algebraic equation (DAE), and it is a generalized formulation of ordinary differential equations (ODE).

4

304

Numerical Solution of Initial and Boundary Value Problems

In some cases, as in (7.64), the DAE system takes the form known as the semiexplicit DAE form given by d y dt 0

=

f1 (t, y, u)

=

f2 (t, y, u)

(7.65)

Assuming we can solve for u explicitly from f2 , that is, 0 = f2 (t, y, u)

→

u = q(t, y)

The DAE is immediately reduced to an ODE set by substitution, that is,   d y = f1 t, y, q (t, y) dt In several instances, however, finding u = q(t, y) explicitly is not easy nor possible. As an alternative approach, we could take the derivative of both equations in (7.65) to yield d2 y dt2

=

∂f1 ∂f1 dy ∂f1 du + + ∂t ∂y dt ∂u dt

0

=

∂f2 ∂f2 dy ∂f2 du + + ∂t ∂y dt ∂u dt

which becomes an ODE set if ∂f2 /∂u remains nonsingular for all t, u, and y. This trick of taking derivatives of a strict DAE set can be done repeatedly until it becomes an ODE set. The minimum number of times a differentiation process is needed for this to occur is known as the index of the DAE. Thus if ∂f2 /∂u remains nonsingular for the system (7.65), it can be classified as an index-1 DAE. Likewise, an ODE set is also known as an index-0 DAE. Several general-purpose DAE solvers are available to solve index-1 DAEs. Most of them are based on the BDF methods. There are also implicit Runge-Kutta methods available. There are specific features of numerical solution of DAEs that distinguish them from that of simple ODEs such as consistency of initial conditions, but we refer to other sources that discuss these issues in great detail.5 Instead, we simply outline the general approach for either implicit Runge-Kutta and multistep BDF methods. Assume that the following DAE set is index-1:   d d F t, y, y = 0 subject to y (t0 ) = y0 and y(0) = z0 (7.66) dt dt The BDF method (generated in Section G.4) results in the following nonlinear equation that needs to be solved for yk+1 :   m 1  F tk+1 , yk+1 , γ(i|k) yk−i =0 (7.67) hk i=−1

where the coefficients γ(|k) are given in (G.22). Because the BDF method is a multistep method, one needs to have values to initiate the recursion. One approach 5

See, for example, K. E. Brenan, S. L. Campbell, and L. R. Petzold, Numerical Solution of InitialValue Problems in Differential-Algebraic Equations, North-Holland, New York, 1989.

7.7 Exercises

305

is to use one-step BDF, followed by two-step and so on, until the maximum number of steps is reached (which has to be less than 7 for stability). This would also mean that to achieve good accuracy at the start of the BDF method, variable step sizes may be needed. For s-stage Runge-Kutta methods, the working equation is given by     s  bj δk, , δk, F =0 j = 1, . . . , s (7.68) (tk + a j h) , yk + =1

yk+1 = yk +

s 

c j δk,

(7.69)

j =1

In most general-purpose DAE programs, such as those found in MATLAB, the form is restricted to the DAEs that can be recast in a mass matrix form, that is, d y = f (t, y) dt where M is known as the mass matrix. If the system is a strict DAE, M will be singular. An example of a DAE problem is given in Exercise E7.14. M (t, y)

7.7 EXERCISES

E7.1. Consider the following linear system   d −20 −5 x= x 10 −35 dt

 x(0) =

1 1



1. Obtain an analytical solution of the system, for example, using techniques in section 6.5.2 or (F.21). 2. Apply the explicit fourth-order Runge-Kutta method and compare the numerical solution with the analytic solution. Use t = 0.01, 0.1, 0.2. 3. Apply the Fehlberg 4/5 embedded Runge-Kutta method (see Section G.5) and compare the numerical solution with the analytic solution. Use t = 0.01, 0.1, 0.2 as the initial time increment, with a tolerance  = 10−8 . 4. Apply the backward Euler method and compare the numerical solution with the analytical solution. Use t = 0.01, 0.1, 0.2. E7.2. Obtain fifth-order versions for Adams-Bashforth, Adams-Moulton, and BDF methods. E7.3. Consider the following time-varying linear system:     d 1 −0.8 − cos (0.5t) 2 x x (0) = x= −3 1 e−0.1t dt 1. Applying the fourth-order BDF method to this linear system, rewrite (7.40) into an explicit equation, that is, determine matrices B0 (t), B1 (t), B2 (t) and B3 (t) such that xk+1 = B0 (t)xk + B1 (t)xk−1 + B2 (t)xk−2 + B3 (t)xk−3 2. Implement the explicit formulas you found in the previous question using

t = 0.01 and compare with the numerical results obtained by using Gauss-Legendre IRK method. (Note: The fourth-order BDF method is a multistep method. You can use the fourth-order explicit Runge-Kutta to initialize the first four terms needed by the BDF method.)

306

Numerical Solution of Initial and Boundary Value Problems

E7.4. One example of a stiff differential system is given by the chemical reaction system known as Robertson’s kinetics: A

→

B

B+B

→

C+B

B+C

→

A+C

and the kinetic equations are given by dCA = −0.04CA + 104 CBCC dt dCB = 0.04CA − 104 CBCC − 3 × 107 C2B dt dCC = 3 × 107 C2B dt Let the initial conditions be: CA(0) = 1, CB(0) = 0 and CC(0) = 0. We want the numerical solution from t = 0 to t = 100. 1. Use the Gauss-Legendre fourth-order IRK to obtain the numerical solution. (Hint: Use an initial time interval of t = 0.01 and a tolerance of  = 1 × 10−8 ). 2. Apply the fixed fourth-order Runge-Kutta method using t = 0.1, 0.01, and 0.001 by simulating from t = 0 to t = 10. Plot the results to note which time increment yields a stable result. 3. Use the MATLAB solver ODE45 to solve the system from t = 0 to t = 10. (Note that the method will appear to “hang.” What is occuring is that the error correction is choosing a very small time increment to avoid the instability caused by the large range of kinetic rates present in the Robertson system.) E7.5. We can use the numerical solution (e.g., the fourth-order explicit RungeKutta) to estimate the parameters of the process. Because the experimental data are sometimes obtained at time instants that are different from those obtained from the numerical solution, an extra step is needed to insert these time instants. For instance, suppose the numerical solution provided the following time sequence: tnum = [0, 0.1, 0.2, 0.3, 0.4, 0.5] If the data were collected at tdata = [0, 0.15, 0.2, 0.34, 0.48] then the numerical solution would need to fill in the additional time instants to be ∗ tnum = [0, 0.1, 0.15, 0.2, 0.3, 0.34, 0.4, 0.48, 0.5]

Let xnum and xdata be the numerical (simulated) values and the data values, respectively, at the time instants given in tdata . Let α be the parameters of the system. Then the parameters can be estimated by performing the following optimization:   αopt = arg min (xnum (α) − xdata ) α

7.7 Exercises

307

Table 7.1. Data for bioprocess Time

x1

x2

Time

x1

x2

Time

x1

x2

0.2419 0.6106 1.0023 1.3940 1.7166

1.0363 1.1018 1.1813 1.2327 1.3076

0.9146 0.8117 0.6573 0.5497 0.4515

2.2235 2.7074 3.3756 4.1590 4.9654

1.3591 1.4199 1.4480 1.4760 1.4854

0.3111 0.2456 0.1942 0.1801 0.1708

6.4171 7.4539 8.3525 9.0438 9.6198

1.5088 1.5088 1.5275 1.5135 1.5368

0.1661 0.1661 0.1661 0.1614 0.1614

For the bioprocess given by the equations, dx1 = (μ − D) x1 dt dx2 μx1 = D (x2f − x2 ) − dt Y where μmax x2 μ= km + x2 Fix D and x2f to be constants. Use an initial guess of D = 0.3 and x2f = 5.0. Also, set initial estimates of the parameters to be: μmax = Y = 0.5 and km = 0.1. Using the optimization described, use the fourth-order Runge Kutta method to estimate the parameters for the least-squares fit of the data given in Table 7.1. Assume x1 (0) = x2 (0) = 1.0. (Hint: Alternatively, you could use the code suggested in oderms.m found in the attached CD-ROM to evaluate the error criterion, and then use minimization routines.) E7.6. Heun’s method is a second-order explicit Runge-Kutta method sometimes written as follows: yk + h f (tk , yk )  h f (tk , yk ) + f (tk+1 , y˜ k+1 ) yk+1 = yk + 2 1. Construct the Runge-Kutta tableau for Heun’s method. 2. Determine the stability region based on the Dalhquist test. y˜ k+1

=

(7.70) (7.71)

E7.7. Consider the following boundary value problem:     d2 y 3 + 2x dy 2 + 4x + x2 + + y=0 dx2 1 + x dx (1 + x)2 subject to y(0) = 1

y(2) = 0.5

Use the shooting method and obtain the plot of y(x) from x = 0 to x = 2. E7.8. Obtain a closed formula for the determinant of the N × N matrix AN given by ⎞ ⎛ 5 −2 0 ⎟ ⎜ .. ⎟ ⎜ −1 . 5 ⎟ ⎜ AN = ⎜ ⎟ . . .. . . −2 ⎠ ⎝ 0 −1 5 (Hint: Follow the method used in Example 7.5.)

308

Numerical Solution of Initial and Boundary Value Problems

E7.9. Obtain the solution for the two-point boundary value problem given by: dCa dt dCb dt dCc dt dCd dt

=

−k1 Ca Cb + k2 Cc − k3 Ca Cc

=

−k1 Ca Cb + k2 Cc

=

k1 Ca Cb − k2 Cc − k3 Ca Cc

=

k3 Ca Cc

subject to the boundary conditions: Cc (10)

=

Cd (10)

Cd (0)

=

0

Ca (10)

=

0.01

Cb(10)

=

0.08

where k1 = 12, k2 = 0.1 and k3 = 2.5. (Hint: Use the initial guess of Ca (0) = Cb(0) = Cc (0) = 0.3 and Cd = 0 ) E7.10. For the fourth-order BDF method given in (7.40), 1. Find the principal root for λ = 0 and the spurious roots. 2. Determine the stability region for the Dahlquist test case. 3. Show that this method is stiffly stable by finding the value of D that would make it D-stable. Is it also Aα -stable? E7.11. Show that the seventh-order BDF method is no longer stable for the Dahlquist test case with λ = 0. (Hint: Solve for the roots and show that some of the spurious roots are greater than 1 when z = λh = 0.) E7.12. Another class of an mth -order implicit multistep method is the Milne-Simpson method for the equation dy/dt = f (y), which is given by the specific form yn+1 = yn−1 + h

m−1 

bj f n−j

j =−1

where f k = f (yk ). Use the same approach as in Section 7.3 (and Section G.3) of using f (y) = y to generate the necessary conditions to show that the matrix equation for the parameter values of bj that is given by ⎛ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎝

⎞⎛

1

1

1

···

1

−1

0

1

···

(m − 1)

.. .

.. .

.. .

..

.. .

(−1)m

0

1

···

.

(m − 1)m

⎞ ⎛ b ⎟ ⎜ −1 ⎟ ⎜ γ1 ⎟⎜ ⎟ ⎜ ⎟⎜ ⎟ ⎜ ⎟⎜ ⎟ ⎜ ⎟ ⎜ b0 ⎟ ⎜ γ2 ⎟⎜ ⎟ ⎜ ⎟⎜ ⎟=⎜ ⎟⎜ . ⎟ ⎜ . ⎟ ⎜ .. ⎟ ⎜ .. ⎟⎜ ⎟ ⎜ ⎟⎜ ⎟ ⎜ ⎠⎝ b ⎠ ⎝ γ m−1 m+1

⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠

7.7 Exercises

where γk =

⎧ ⎪ ⎪ ⎨ 0 2 ⎪ ⎪ ⎩ k

309

if k is even if k is odd

Thus show that for m = 4, we have  h  yn+1 = yn−1 + 29f n+1 + 124f n + 24f n−1 + 4f n−2 − f n−3 90 E7.13. Determine the conditions under which the linear two-point boundary conditions have a unique solution. Under what conditions will the two-point boundary value problems have no solution? E7.14. Consider the following set of DAE equations describing the equilibrium flow rates for a catalytic reaction, A → B + C, in a one-dimensional flow through a packed column, assuming ideal gas behavior,6     dn˙ A dn˙ B dn˙ C P n˙ C − = = = kr · cAs − cBs ds ds ds Ks n˙ tot dP ds

=

0

=

0

=

ψ n˙ tot RT ˙ P Vin   n˙ A cAs − KaAP (ctot − cAs − cBs ) n˙ tot   n˙ B cBs − KaBP (ctot − cAs − cBs ) n˙ tot −

where the states n˙ A, n˙ B, and P are the molar flow rate of A, molar flow rate of B, and pressure, respectively, at a point where the total weight of catalyst away from the inlet is given by s. The total molar flow is n˙ tot = n˙ A + n˙ B + n˙ C + n˙ G , where n˙ G is the molar flow rate of an inert gas. The variable V˙ in is the volumetric flow rate at the inlet. The other variables cAs and cBs are the adsorbed concentration of A and B per unit mass of catalyst. The total number of catalytic sites is assumed to be a constant ctot . The parameter ψ results from using the Ergun equation; kr is the specific rate of reaction at the sites; and Ks , KaA, and KaB are equilibrium constants for the reaction, adsorption of A, and adsorption of B, respectively. Finally, R = 8.3145 Pa · m3 (mole K)−1 is the universal gas constant. Assume the following parameter set and inlet conditions: ψ = 11.66

6

Pa kg Cat

ctot = 10−3

moles sites kg Cat

kr = 30

moles sec−1 moles sites

KaA = 4.2 × 10−5 Pa−1

KaB = 2.5 × 10−5 Pa−1

Ks = 9.12 × 105 Pa

T in = T = 373 K

m3 V˙ in = 0.001 sec

Pin = 1 atm

p A,in = 0.1 atm

p B,in = p C,in = 0 atm

Based on an example from K. Beers, Numerical Methods for Chemical Engineering, Cambridge University Press, Cambridge, UK, 2007.

310

Numerical Solution of Initial and Boundary Value Problems

Based on ideal gas behavior, the initial conditions are: P(0) = Pin n˙ A(0) =

p A,in V˙ in RT in

n˙ B(0) = n˙ C(0) = 0 n˙ G (0) =

(Pin − p A,in ) V˙ in = n˙ G RT in

1. Reduce the DAE to an ODE by first solving for cAs and cBs in terms of n˙ tot and P, and then substitute the results into the differential equations. Solve the ODE using the available ODE solvers (e.g., in MATLAB). 2. By converting it to a DAE using the mass matrix form, solve the DAE directly using available DAE solvers (e.g., in MATLAB). Compare the solution with that obtained from the previous method.

8

Qualitative Analysis of Ordinary Differential Equations

In some applications, the qualitative behavior of the solution, rather than the explicit solution, is of interest. For instance, one could be interested in the determination of whether operating at an equilibrium point is stable or not. In most cases, we may want to see how the different solutions together form a portrait of the behavior around particular neighborhoods of interest. The portraits can show how different points such as sources, sinks, or saddles are interacting to affect neighboring solutions. For most scientific applications, a better understanding of a process requires the larger portrait, including how they would change with variations in critical parameters. We begin this chapter with a brief summary on the existence and uniqueness of solutions to differential equations. Then we define and discuss the equilibrium points of autonomous sets of differential equations, because these points determine the sinks, sources, or saddles in the solution domain. Next, we explain some of the technical terms, such as integral curves, flows, and trajectories, which are used to define different types of stability around equilibrium points. Specifically, we have Lyapunov stability, quasi-asymptotic stability, and asymptotic stability. We then briefly investigate the various types of behavior available for a linear second-order system, dx/dt = Ax, A[=]2 × 2, for example, nodes, focus, and centers. Using the tools provided in previous chapters, we end up with a convenient map that relates the different types of behavior, stable or unstable, to the trace and determinant of A. Afterward, we discuss the use of linearization to assess the type of stability around the equilibrium points. However, this approach only applies to equilibrium points whose linearized eigenvalues have real parts that are non-zero. For the rest of the cases, we turn to the use of Lyapunov functions. These are functions that are often related to system energy, yielding a sufficient condition for asymptotic stability. The main issue with Lyapunov functions, however powerful and general, is that there are no general guaranteed methods for finding them. Next, we move our attention to limit cycles. These are special periodic trajectories that are isolated; that is, points nearby are ultimately trapped in the cycle. Some important oscillators such as van der Pol equations exhibit this behavior. Two theorems, namely Bendixson and Poincare´ -Bendixson’s theorems, are available for existence (or nonexistence) of limit cycle in a given region. Another important tool is the Poincare´ map, which transforms the analysis to a discrete transition maps. We explore the use of Poincare´ maps together with Lyapunov analysis to show the 311

312

Qualitative Analysis of Ordinary Differential Equations

existence and uniqueness of a limit cycle for a class of nonlinear systems known as ` the Lienard system. We also include a discussion of nonlinear centers, because these are also periodic trajectories but are not isolated; thus they are not limit cycles. A brief discussion on bifurcation analysis is also included in Section H.1 as an appendix. These analyses are aimed at how the phase portraits (i.e., collective behavior of the system) are affected, as some of the parameters are varied. It could mean the addition or loss of equilibrium points or limit cycles, as well as changes in their stabilities. Qualitative analysis of dynamical systems encompasses many other tools and topics that we do not discuss, such as nonautonomous systems and chaos.

8.1 Existence and Uniqueness Before we characterize the solutions around neighborhoods of initial points x0 , we need the conditions for the existence and uniqueness of solutions of a differential system. These conditions are given in the following theorem: THEOREM 8.1.

For a system given by dx = f (t, x) dt

If f is continuous and continuously differentiable in t and x around t = t0 and x = x0 , then there exists a unique solution x (t) in a neighborhood around t = t0 and x = x0 where the conditions of continuity are satisfied. This theorem is well established.1 We can sketch the proof as follows: 1. The conditions for the continuity of the partial derivatives ∂f/∂t and ∂f/∂x, together with the fundamental theorems of calculus, imply that for every (t, x) ∈ R, where R is a region where f (t, x) is continuous and continuously differentiable, then there is a constant K > 0 such that f (t, x) ˆ − f (t, x) ≤ K xˆ − x

where x, ˆ x∈R

These conditions are also known as Lipschitz conditions. 2. To establish the existence of a solution, Picard’s iterative solution is used to find the kth approximate solution, xk  t xk (t) = x0 + f (τ, xk−1 ) dτ 0

which yields  xk (t) − xk−1 (t) =

t

[f (τ, xk−1 ) − f (τ, xk−2 )] dτ

0

The Lipschitz conditions are then used to show that Picard’s iteration is convergent to the solution x (t), thus showing the existence of a solution. 1

See, for example, A. C. King, J. Billingham, and S. R. Otto, Differential Equations: Linear, Nonlinear, Ordinary and Partial, Cambridge University Press, UK, 2003.

8.2 Autonomous Systems and Equilibrium Points

3. To establish uniqueness, let y(t) and z(t) be two solutions, that is,  t  t y(t) = x0 + f (s, y) ds and z(t) = x0 + f (s, z) ds t0

t0

or



t

y(t) − z(t) =

[f (s, y) − f (s, z)] ds

t0

Taking the norm of the left-hand side and applying the Lipschitz condition, we get  t  t f (s, y) − f (s, z) ds ≤ K y(s) − z(s) ds y(t) − z(t) ≤ t0

t0

Finally, another lemma known as Gronwall’s inequality is used. The lemma states that given L ≤ 0 and non-negative h(t) and g(t) for t ∈ [a, b], then  t h(t) ≤ L + h(s)g(s)ds a

implies



t

h(t) ≤ L exp

 g(s)ds

a

This lemma can then be applied by setting h(t) = y − z , g(t) = K and L = 0 to yield y(t) − z(t) = 0 Thus the conditions of continuity for both f and its partial derivatives are also sufficient to yield a unique solution. Although the existence and uniqueness theorem is a local result, that is, in the neighborhood of the initial point x0 , this region can be as large as the conditions allow, for example, in a region where the continuity of both f and its derivatives are satisfied. A discontinuity in f(t, x) is necessary for non-uniqueness. A simple example of such a case is given in Exercise E8.1. This implies that when one needs to model a dynamic system by patching together different regions, it may be worthwhile to ensure smoothness in f; that is, using cubic splines instead of linear interpolations guarantees unique solutions of the models.

8.2 Autonomous Systems and Equilibrium Points Recall that the function f(t, x) = f(x) (i.e., it is not an explicit function of t), the system dx = f(x) (8.1) dt is autonomous. Otherwise it is known as nonautonomous. We limit our discussions in this chapter only to autonomous systems. A point xe is an equilibrium point (also known as fixed point or stationary point) of (8.1) if f(xe ) = 0 for t ≥ 0. Because f (x) is generally a set of nonlinear equations, there may be more than one equilibrium point. In some cases, where xe has to be

313

314

Qualitative Analysis of Ordinary Differential Equations

real-valued and finite, there may even be no equilibrium points. The presence of multiple, isolated equilibrium points is a special feature of nonlinear systems.2

EXAMPLE 8.1.

The equilibrium points of the following system: dx1 = x2 dt

and

are given by x1e =

−b ±

dx2 = ax21 + bx1 + c dt

√

b2 − 4ac 2a

and

x2e = 0

If b2 < 4ac, the values for x1e are complex numbers. Thus if the x is constrained to be real-valued, we say that no equilibrium points exist for this case. However, if b2 = 4ac, we see that the only equilibrium point is for x2e = 0 and x1e = −b/(2a). Finally, if b2 > 4ac, we have two possible equilibrium points, ⎛ ⎞ ⎛ ⎞ √ √ −b − b2 − 4ac −b + b2 − 4ac ⎜ ⎟ ⎜ ⎟ 2a 2a ⎟ ⎟ and [xe ]1 = ⎜ [xe ]2 = ⎜ ⎝ ⎠ ⎝ ⎠ 0 0 Note that in most cases, numerical methods such as the Newton method given in Section 2.9 may be needed to find the equilibrium points.

Remark: One of the tricks used in Chapter 7 to handle nonautonomous systems via numerical solvers of initial value problems was to extend the state x by adding ˆ This is accompanied by extending f with f n+1 = 1 another state xn+1 = t to form x. ˆ This method might be appealing because one might argue that we get to form f. back an autonomous system. However, doing so immediately presents a problem ˆ Thus if one applies the state extension approach because with f n+1 = 1, fˆ = 0 for all x. to analyze a nonautonomous system, the definition of equilibrium points or other solutions such as periodic solutions will have to be modified to apply strictly to the original state vector x, that is, without the time-variable, xn+1 = t.

8.3 Integral Curves, Phase Space, Flows, and Trajectories Assuming that (8.1) can be solved around the neighborhood of initial conditions, x0 ∈ D, where D is an open set in Rn , we can represent the solutions of (8.1) by integral curves, C(t) of (8.1), which are simply a set of curves whose tangents are specified by f(x), that is,   (8.2) C(t) = x1 (t), . . . , xn (t) and

2

⎞ f 1 (x1 , . . . , xN ) d ⎟ ⎜ .. C(t) = ⎝ ⎠ . dt f n (x1 , . . . , xN ) ⎛

(8.3)

We need the descriptor “isolated” because linear systems can also have multiple equilibrium points but they would not be isolated.

8.3 Integral Curves, Phase Space, Flows, and Trajectories 1

315

1.5

x y 1

0.5

0.5

x, y

y

0

0

−0.5

−1 0

−0.5

5

10

15

20

25

30

−1 −1

−0.5

t

0

0.5

1

x

Figure 8.1. On the left is the plot of solutions x and y as functions of t. On the right is the integral curve shown in the phase plane (with the initial point shown as an open circle).

This appears to be redundant because C(t) is nothing but x(t) that satisfies (8.1). One reason for creating another descriptor such as “integral curves” is to stress the geometric character of the curves C(t). For instance, an immediate consequence is that because C(t) are simply curves parameterized by t, we can analyze and visualize the system behavior in a space involving only the components of x, that is, without the explicit information introduced by t. We refer to the space spanned by the integral curves (i.e., spanned by x1 , . . . , xn ) as the phase space, and the analysis of integral curves in this space is also known as phase-space analysis. For the special case of a two-dimensional plane, we call it a phase-plane analysis.

EXAMPLE 8.2.

Consider the autonomous system given by

dx dy = y and = −1.04x − 0.4y dt dt There is only a single equilibrium point, which is at the origin. For the initial condition at x0 = (1, 0)T , the solutions x(t) and y(t) are plotted together in Figure 8.1 as functions of t. Also included in Figure 8.1 is the phase-plane plot of y versus x of the integral curve starting at (x, y) = (1, 0). When exploring the solutions starting at different initial conditions, the advantage of using phasespace plot becomes clearer – various integral curves can be shown together in one figure.

Another consequence of the concept of integral curves is that the curves can have parameterizations other than t, for example, if f 1 = 0, the curves can be described by dx2 f2 = , dx1 f1

...,

dxn fn = dx1 f1

(8.4)

The solution of (8.4) should yield the same curves. The equations in (8.4), or the original set given in (8.1), may not be easy to integrate analytically. In general, numerical IVP (initial value problem) solvers discussed in Chapter 7 are needed. As a supplementary approach, (8.4) suggests that

1.5

316

Qualitative Analysis of Ordinary Differential Equations

the slopes (independent of t) can present visual cues to the shapes of the integral curves. This leads us to the use of direction field plots. A direction field, which we  , is a vector field that gives the slopes of the tangents of the integral curves in denote d phase space. It is not the same as the velocity field, because the vectors in a direction field have the same magnitudes at all points except at the equilibrium points. The components of a direction field for a given f (x) can be obtained as ⎧ ⎪ ⎪ ⎪ ⎪ 0 if f i = 0 ⎪ ⎨ (8.5) di (x) = ⎪ f i (x) ⎪ ⎪ ⎪ otherwise α ⎪ ⎩ f (x) where α is a scaling constant chosen based on visual aspects of the field. In the formulation in (8.5), the equilibrium points are associated with points rather than vectors to avoid division by zero.3 The main advantage of direction fields is that the formulas given in (8.5) are often much easier to evaluate. The direction fields are often evaluated at points specified by rectangular, cylindrical, or spherical meshes. Furthermore, one could collect the locus of points having the same slopes to form another set of curves known as isoclines. A special case of isoclines are those that collect points with slopes that are zero in one of the dimensions, and these are known as nullclines. For instance, for the 2D case and rectangular coordinates (x, y), the nullclines are the lines where the x components or y components are zero. Alternatively, for the 2D case under polar coordinates (r, θ), the nullclines are the lines where the slopes are radially inward or outward (i.e., no angular components) or those where the slopes are purely angular (i.e., no radial components).

EXAMPLE 8.3.

Take the same system given in Example 8.2,

dx dy = y and = −1.04x − 0.4y dt dt Then the direction field at a rectangular grid around the origin is shown in Figure 8.2. Also, there are four nullclines shown in the right plot in Figure 8.2.

In addition to integral curves, we need another concept, known as flows. We use flows during our definition of stability of equilibrium points. Definition 8.1. The flow of (8.1) is a mapping ϕ : Rn × R → Rn , such that 1. The derivative with respect to t are given by ∂ ϕ (x, t) = f (ϕ (x, t)) ∂t

(8.6)

2. ϕ (x, 0) = x 3. ϕ (x, s + t) = ϕ (ϕ(x, s), t) Essentially, flows are the mechanism by which the path of an integral curve can be traversed. Thus flows specify the forward or backward movements at a 3

In the terminology of direction fields, equilibrium points are called singular points, whereas the rest are called regular points.

8.4 Lyapunov and Asymptotic Stability 1.5

1.5

1

1

0.5

317

0.5

y

y 0

0

−0.5

−0.5

−1 −1

−0.5

0

0.5

1

1.5

−1

−1

x

−0.5

0

0.5

1

x

Figure 8.2. The figure on the left shows the direction field for Example 8.3, whereas the figure on the right shows the four nullclines under the rectangular coordinates as dotted lines, that is, locus of purely left, purely right, purely up, and purely down slopes.

specified point in the phase space, thereby yielding a definite direction in the movement along the paths. In this respect, integral curves equipped with flows are called trajectories.4

8.4 Lyapunov and Asymptotic Stability There are several concepts and types of stability of equilibrium points. These include Lyapunov stability, quasi-asymptotic stability, and asymptotic stability. Definition 8.2. An equilibrium point xe of (8.1) is Lyapunov stable if for every  > 0 there exists a δ > 0 such that if xe − y < δ then ϕ(xe , t) − ϕ(y, t) <  for t ≥ 0, where ϕ(x, t) is the flow of (8.1) at the point x. Lyapunov stability means that if a point y is close to the equilibrium point xe , then the flow originating from both x and y will remain close, as shown in Figure 8.3. Another type of stability of an equilibrium point xe is quasi-asymptotic stability (also known as attractive property). Definition 8.3. An equilibrium point xe of (8.1) is quasi-asymptotically stable if there exists a δ > 0 such that if |xe − y| < δ then   lim ϕ(xe , t) − ϕ(y, t) = 0

t→∞

Unlike Lyapunov stability, a quasi-asymptotic stability does not allow arbitrary specification of  > 0 that bounds ϕ(y, t) − xe . Instead, all it requires is that eventually the distance will converge to zero, as shown in Figure 8.4. 4

In most texts, there is no distinction between integral curves and trajectories. However, because we have claimed that integral curves can be considered simply as curves in the phase space, they can be reparameterized also by −t for autonomous systems. Thus we suggest that the term “trajectories” is a more appropriate term when the direction of the path as t increases is important.

1.5

318

Qualitative Analysis of Ordinary Differential Equations

y

Figure 8.3. Lyapunov stability around the equilibrium point, xe .

x

Some equilibrium points can be Lyapunov stable and not quasi-asymptotic (see, e.g., Exercise E8.2), whereas others can be quasi-asymptotically stable but not Lyapunov stable. An example of a system that is quasi-asymptotically stable but not Lyapunov stable is the Vinograd system described in the following example.

A system developed by R. E. Vinograd is given by the following pair of nonlinear autonomous differential equations:

EXAMPLE 8.4.

dx dt

=

f 1 (x, y) =

x2 (y − x) + y5   2 (x2 + y2 ) 1 + (x2 + y2 )

dy dt

=

f 2 (x, y) =

y2 (y − 2x)   2 (x2 + y2 ) 1 + (x2 + y2 )

(8.7)

The only equilibrium point is the origin. A plot of the direction field for the Vinograd system (8.7) is shown in Figure 8.5. As an alternative, we can represent the system in polar coordinates, that is, dr = fˆ r (r, θ) dt

and

dθ = fˆ θ (r, θ) dt

r3 h1 (θ) + rh2 (θ) fˆ r = 1 + r4

and

r2 h3 (θ) + h4 (θ) fˆ θ = 1 + r4

(8.8)

where

with h1 (θ)

=

h2 (θ)

=

h3 (θ)

=

h4 (θ)

=

sin θ cos θ (cos θ − 1)2 (cos θ + 1)2     sin θ cos θ 3 cos2 θ − 2 + 1 − 2 cos2 θ [(cos θ − 1) (cos θ + 1)]3   cos θ sin θ − 3 cos θ + 3 cos3 θ

Note that limr→0 ( fˆ r /r) = h2 (θ) and limr→0 fˆ θ = h4 (θ).

y

x

Figure 8.4. Quasi-asymptotic stability around x, an equilibrium point.

8.4 Lyapunov and Asymptotic Stability

319

2

1.5

1

0.5

y Figure 8.5. The direction field for the Vinograd system given in (8.7).

0

−0.5

−1

−1.5

−2

−2

−1

0

1

x

Using the nullclines of (8.8), we can find different sectors where fˆ r and fˆ θ change signs, as shown in Figure 8.6. In the figure, we have the shaded region where fˆ r > 0, and the unshaded region as fˆ r < 0, with fˆ r = 0 at the boundaries. However, inside the regions ABDCA, AJKIA, AGHA, and AEFA, we have fˆ θ > 0, and outside these regions, fˆ θ < 0. Along the curves ABD, ACD, AIK, and AJK, the tangents to the trajectories are pointing radially outward (i.e., vθ = 0), whereas along the curves AG, AH, AF , and AE, the tangent to the trajectories is pointing radially inward. This shows that local trajectories northwest of GACD or southeast of FAIK will be repelled from the origin, whereas the other regions will be attracted to the origin. The trajectories for the Vinograd system starting at different initial points can also be obtained by numerical evaluation of (8.7) using the IVP solvers such as Runge-Kutta methods given in the previous chapter. These are shown in Figure 8.7.5 The plot in Figure 8.7 is consistent with both Figures 8.5 and 8.6. It shows that initial conditions starting in some locations of the (x, y)-plane will go directly to the origin, whereas starting at other locations may initially diverge away from the origin but ultimately will converge to the origin. This is an example of a case in which the equilibrium point is quasi-asymptotically stable but not Lyapunov stable.

The third type of stability is asymptotic stability. Definition 8.4. An equilibrium point xe of (8.1) is asymptotically stable if it is both Lyapunov stable and quasi-asymptotic stable. Figure 8.8 shows the property of asymptotic stability around x. The equilibrium point for the system given in Example 8.2 is asymptotically stable. 5

Because the equilibrium point is both unstable and attractive, the numerical round-off errors may produce artificial errors, possibly showing apparent chaotic behavior. A simple fix is to provide smaller error tolerance and also setting the values of the derivative functions f 1 (x, y) and f 2 (x, y) to be zero if they are within the chosen error tolerance.

2

320

Qualitative Analysis of Ordinary Differential Equations

D

0.3 (+)

B

C

E (+)

y 0

G

F

A

(+)

H

I

Figure 8.6. Different regions based on whether vr and vθ are positive or negative. Shaded region shows where vr > 0, whereas regions marked by (+) show where vθ > 0.

J (+)

K

-0.3 -0.3

0.3

0

x

Asymptotic stability is a much stronger type of stability than either Lyapunov stability or quasi-asymptotic stability, and therefore, it is the more desirable type of stability for several physical processes. It specifies that the flows of neighboring points around the equilibrium point can be bounded, and in addition, they will ultimately reach the equilibrium point. Often, in engineering systems such as in a manufacturing process, a chosen steady state is often considered to be a target based on optimization between productivity and cost. Controllers are often attached to the system to ensure that these targets will be achieved and will stay put. Thus being able to maintain stability is often the key to sustained operation. Although asymptotic stability is often desired, in some cases, bounding the states to within a satisfactory neighborhood around the target is sufficient for practical purposes (i.e., Lyapunov stability is all that is needed). Indeed, stability analysis is often the main impetus for using a qualitative (or descriptive) analysis of most engineering systems. If necessary, one can

2

1.5

1

0.5

y

Figure 8.7. Different trajectories of the Vinograd system using different initial points shown by the open circles.

0

−0.5

−1

−1.5

−2

−2

−1

0

x

1

2

8.5 Phase-Plane Analysis of Linear Second-Order Autonomous Systems

Figure 8.8. Asymptotic stability around x, an equilibrium point.

321

y

x

engineer additional components and procedures such as feedback control to achieve the required behavior. In the next two sections, we discuss the phase-plane analysis of a linear second order and the linearization approximations of nonlinear systems. The stability analysis of these approaches can be achieved using eigenvalue analysis and is often used to determine asymptotic stability. However, the linearization approach is only limited to situations in which the eigenvalues do not contain zeros or pure imaginary values. We return to Lyapunov and aysmptotic stability of nonlinear systems after these two sections, and we discuss a more general (albeit sometimes elusive) approach of using positive definite functions known as Lyapunov functions.

8.5 Phase-Plane Analysis of Linear Second-Order Autonomous Systems In this section, we obtain different phase-plane portraits of linear autonomous second-order systems given by   d a11 a12 x = Ax = x (8.9) a21 a22 dt Aside from having a better grasp of the different types of possible behaviors based on A, these results will remain useful for several second-order nonlinear systems because the linearized approximation of their trajectories will be sufficiently close to the original nonlinear forms, at least around the equilibrium points. The characteristic polynomial, eigenvalues, and eigenvectors can be found in terms of the trace and determinant of A as given in (3.20), (3.21), and (3.22), which are repeated below: Characteristic polynomial: Eigenvalues:

Eigenvectors:

λ2 − tr(A)λ + det(A) = 0 ! tr(A) ± tr(A)2 − 4det(A) λ= 2 ⎧ ⎪ ⎛ ⎞ ⎪ ⎪ ⎪ a ⎪ 12 ⎪ ⎪ ⎜ ⎟ ⎪ ⎪ ⎜ λ∗ − a11 ⎟ if a11 = λ∗ ⎪ ⎪ ⎝ ⎠ ⎪ ⎪ ⎪ ⎪ ⎪ 1 ⎪ ⎨ vλ∗ = ⎪ ⎛ ⎞ ⎪ ⎪ ⎪ 1 ⎪ ⎪ ⎪ ⎜ ⎟ ⎪ ⎪ a21 ⎜ ⎟ if a22 = λ∗ ⎪ ⎪ ⎝ ⎠ ⎪ ⎪ ⎪ λ − a ∗ 22 ⎪ ⎪ ⎪ ⎩

322

x2

Qualitative Analysis of Ordinary Differential Equations

0.8

0.8

0.4

0.4

x2

0.0

-0.4

0.0

-0.4

-0.8

-0.8

-1.0

-0.5

0.0

0.5

1.0

-1.0

-0.5

0.0

x1

0.5

1.0

x1

Figure 8.9. The right figure shows the trajectories around a stable node with 0 > λ1 > λ2 . The left figure shows the trajectories around an unstable node with λ1 > λ2 > 0.

There are only three possible cases when λ∗ = a11 = a22 – either A is strictly diagonal, upper triangular, or lower triangular, all of which have λ in the diagonals. In the strictly diagonal case, the eigenvectors are e1 and e2 . For the triangular cases, there is only one linearly independent eigenvector: e1 for the upper triangular case, and e2 for the lower triangular case. When A is nonsingular, the origin is the only equilibrium point. If A is singular and of rank 1, then the equilibrium points will lie in a line containing the eigenvector that corresponds to λ = 0. Lastly, when A = 0, the set of all equilibrium points is the whole space; that is, no motion occurs. Let λ1 and λ2 be the eigenvalues of A. If the eigenvalues are both real, then the trajectories are classified as either nodes, stars, improper nodes, saddles, or degenerate. Otherwise, if the eigenvalues are complex-valued, the trajectories are either focuses or centers, where centers occur when the eigenvalues are pure imaginary. We discuss each case next. 1. Nodes. When det(A) > 0 and tr(A)2 > 4 det(A), then both λ1 and λ2 are realvalued and have the same sign. Both eigenvalues are negative when tr(A) < 0, and both are positive when tr(A) > 0. In either case, using the diagonalization procedure of Section 6.6, the solution of (8.9) is given by     (8.10) x(t) = z10 eλ1 t v1 + z20 eλ2 t v2 where



z10 z20

 =



v1

v2

−1

x0

If both eigenvalues are positive, the equilibrium points are classified as unstable nodes. Otherwise, if both eigenvalues are negative, the equilibrium points are stable nodes. Based on (8.10), the trajectories x(t) are linear combinations of the eigenvectors v1 and v2 . If the initial point x0 happens to be along either of the eigenvectors, then the trajectories will travel along the same line as that contains the eigenvectors. Otherwise, the trajectories will be “half-U-shaped,” where the center of the “U” is along the eigenvector that corresponds to the eigenvalue with the larger absolute value. Typical plots of both stable and unstable nodes are shown in Figure 8.9.

8.5 Phase-Plane Analysis of Linear Second-Order Autonomous Systems

1.0

x2

0.0

-1.0 -1.0

0.0

1.0

x1 Figure 8.10. The trajectory around a saddle with λ1 > 0 > λ2 where line 1 is along v1 and line 2 is along v2 .

2. Saddles. When det(A) < 0, the eigenvalues are both real-valued, but one of them will be positive, whereas the other will be negative. Thus let λ1 > 0 > λ2 ; then, based on (8.10), x(t) will be a linear combination of an unstable growth along v1 and a stable decay along v2 . The equilibrium points will then be classified as saddles. Typical plots of trajectories surrounding saddles are shown in Figure 8.10. 3. Stars. When A = a I, with a = 0, both eigenvalues  will be equal to a, and the matrix of eigenvectors becomes V = v1 v2 = I. This further implies z = V x0 = x0 . Thus (8.10) reduces to x(t) = eat x0

(8.11)

and the trajectories will follow along the vector x0 . The equilibrium points are then classified as stars, and depending on a, they could be stable stars (if a < 0) or unstable stars (if a > 0). Typical plots of trajectories surrounding stars are shown in Figure 8.11. 4. Improper Nodes. Suppose the eigenvalues of A are equal to each other, that is, λ1 = λ2 = λ = 0. Using the method of finite sums (cf. 6.14) to solve this case with repeated roots, the solution is given by x(t) = eλt [I + t (A − λI)] x0

(8.12)

Note that if A = λI, we get back the equation of the trajectories around that of a star node, as given in (8.11). Thus add another condition that A is not diagonal. In this case, there will be only one eigenvector, which is given by     λ − a22 a12 or v= v= λ − a11 a21 whichever is nontrivial. If the initial point x0 = αv (i.e., it lies along the line containing v), then the trajectories will travel along that line because (A − λI)v = 0 and (8.12) becomes x(t) = αeλ v. If x0 is outside of this line, the trajectories will be curved either

323

324

x2

Qualitative Analysis of Ordinary Differential Equations

0.8

0.8

0.4

0.4

x2

0.0

-0.4

0.0

-0.4

-0.8

-0.8

-1.0

-0.5

0.0

0.5

1.0

-1.0

-0.5

x1

0.0

0.5

1.0

x1

Figure 8.11. The trajectories surrounding (a) stable stars and (b) unstable stars.

with a half-S-shape (S-type for short) or with a reverse-half-S-shape (Z-type for short). In either case, the equilibrium points for this case are classified as improper nodes. Again, stability will depend on λ: stable if λ < 0 and unstable if λ > 0. Typical plots of the trajectories surrounding both types of stable improper nodes are shown in Figure 8.12. To determine whether the improper node is S-type or Z-type, one can show (see, e.g., Exercise E8.5) that the following conditions can be used: S-type:

a12 λ > 0

;

as long as

a12 = 0

or a21 λ < 0

;

as long as

a21 = 0

a12 λ < 0

;

as long as

a12 = 0

or a21 λ > 0

;

as long as

a21 = 0

If

Z-type: If

(8.13)

5. Focus and Centers. When tr(A) < 4 det(A), the eigenvalues of A are a complex conjugate pair given by λ1 = ρ + iω and λ2 = ρ − iω, where ρ

=

ω

=

tr(A) 2 " 1 4 det(A) − tr(A)2 2

(8.14)

Figure 8.12. Trajectories surrounding stable (a) S-type and (b) Z-type improper nodes.

8.5 Phase-Plane Analysis of Linear Second-Order Autonomous Systems 4

principal line 2

2 4

x2

principal line 1

0

x2

0

-2 -4

-4 -4

bounding ellipse

-2

0

2

4

-4

x1

0

4

x1

Figure 8.13. The left figure shows a stable focus and the right figure shows an unstable focus. Included in both figures are the bounding ellipses. The principal lines are the polar nullclines, where the points of these lines have no radial components.

Alternatively, a necessaryand sufficient condition for eigenvalues to have nonzero imaginary parts is (a11 − a22 )2 + 4a12 a21 < 0 . Based on (8.10), the solution can be simplified in terms of trigonometric functions to be 8  9 sin(ωt) 2a12 a11 − a22 ρt x(t) = e cos (ωt) I + x0 (8.15) 2a21 a22 − a11 2ω When ρ = 0, the responses are periodic, and the trajectories in the phase plane will be ellipses centered at the equilibrium point, and thus the equilibrium point is known as a center. However, if ρ = 0, the trajectories will be elliptical spirals, and the equilibrium point is known as a focus. When ρ < 0, the spiral moves toward the equilibrium point, which is then called a stable focus, and when ρ > 0, the spiral moves away from the equilibrium point, which is then called an unstable focus. Two more pieces of information can be determined from A: the direction of the rotation and the bounding ellipse. Surprisingly, the rotation depends only on the sign of a12 . It can be shown that the trajectories around centers and focuses will be clockwise if a12 > 0 and counterclockwise if a12 < 0 (see Exercise E8.4). The bounding ellipse of a stable (unstable) focus is the minimal (maximal) ellipse that contains the initial point of which the trajectory will not cross again. The points of the bounding ellipse x¯ can be found as 8 x¯ (ψ) =

1

9

cos (ψ) I + sin (ψ) ! A x0 det(A)

;

0 ≤ ψ ≤ 2π

(8.16)

Typical plots of the trajectories surrounding stable and unstable focuses are shown in Figure 8.13, whereas the trajectories surrounding centers are shown in Figure 8.14.

325

326

Qualitative Analysis of Ordinary Differential Equations

4

x2

0

principal

Figure 8.14. Trajectories surround a center.

principal

-4 -4

0

4

x1 6. Degenerate Points. When one or both eigenvalues of A are zero, there will be more than one equilibrium point. Both these cases are classified as degenerate points or nonisolated equilibrium. Let λ1 = 0 and λ2 = 0; then (8.10) will reduce to x = z01 v1 + z02 eλ2 t v2

(8.17)

This means that when x0 = αv1 then x(t) = αv1 ; that is, v1 will lie in the line that contains all the (nonisolated, non-unique) equilibrium points. Outside this line, the trajectories are parallel to v2 , that is, an affine operation on v2 . Again, the equilibrium points will be stable if λ2 < 0 and unstable if λ2 > 0. Typical plots of trajectories surrounding degenerate equilibrium points are shown in Figure 8.15. If both eigenvalues happen to be equal to zero, then we must have both the trace and determinant of A to be zero. This implies a11 = −a22 and a12 a21 = −a211 . From (8.12), we have x(t) = (I + At) x0

equilibrium line

equilibrium line

eigenvector

eigenvector

Figure 8.15. Trajectories surrounding (a) stable and (b) unstable degenerate points.

8.6 Linearization Around Equilibrium Points

327

det(A) 2

FOCUS (stable)

FOCUS (unstable)

NODES (stable)

NODES (unstable)

IMPROPER

CENTER

STARS

Trace(A)

SADDLE

Figure 8.16. The regions containing different types of trajectories.

If A = 0, no motion is present. If A = 0 with λ1 = λ2 = 0, then no motion is present along the line containing the eigenvector v, where  ⎧  a12 ⎪ ⎪ if a12 = 0 ⎪ ⎪ −a11 ⎨ v=   ⎪ ⎪ ⎪ −a22 ⎪ ⎩ if a21 = 0 a21 Outside this line, the trajectories are along straight lines that are parallel to v. The trajectories for A = 0 are left as Exercise E8.6. The different types of equilibrium points can be summarized in terms of the trace and determinant of A, as shown in Figure 8.16.

8.6 Linearization Around Equilibrium Points Let xe be an equilibrium point of the autonomous system described by dx/dt = f(x); then by expanding f using Taylor series around x = xe , we obtain  df  f (x) = f (xe ) + (8.18) (x − xe ) + O(x − xe 2 ) dx x=xe  df  Also, let J (xe ) = , that is, the Jacobian matrix at the equilibrium point xe . dx x=xe After truncating the higher order terms, O(x − xe 2 ), together with the fact that because f (xe ) = 0, we have f (x) ≈ J (xe ) (x − xe )

(8.19)

328

Qualitative Analysis of Ordinary Differential Equations

which then yields a linearized approximation of the original nonlinear system given by d x = J (xe ) (x − xe ) dt

(8.20)

However, this approximation is true only for small neighborhoods around the equilibrium points. Moreover, if the Jacobian is singular (i.e., it contains some zero eigenvalues) or if it contains pure imaginary eigenvalues, then the truncations may no longer be valid, even for a small neighborhood around the equilibrium points. Thus we need to classify the condition for which linearization would be sufficiently close to the actual flows around the equilibrium points, as given by the following definition: Definition 8.5. An equilibrium point xe of dx/dt = f(x) is classified as a hyperbolic equilibrium point if none of the eigenvalues of the Jacobian matrix J (xe ) of f(x) are zero or pure imaginary. Otherwise, it is classified as a non-hyperbolic equilibrium point. The following theorem, known as the Hartman-Grobman theorem or the linearization theorem, applies only to hyperbolic equilibrium points: Let xe be a hyperbolic equilibrium point of dx/dt = f(x). Then for a small neighborhood around xe , the behavior of the trajectories can be approximated by (8.20).

THEOREM 8.2.

This implies that the type and stability of the trajectories surrounding hyperbolic equilibrium points can be determined by simply analyzing the linearized equations.

EXAMPLE 8.5.

Consider the following second-order system     d x2 x1 = x2 cos (πx1 ) − x2 dt

(8.21)

The equilibrium points and the corresponding Jacobian matrices at these points are given by     0 1 n − 21 and J (xe )n = (xe )n = (−1)n π −1 0 where n is an integer. The characteristic equation is λ2 + λ − (−1)n π = 0, and the eigenvalues are " 1 (i)n λ=− ± 4π + (−1)n 2 2 √ where i = −1. Thus all the equilibrium points are hyperbolic, and we can use linearized approximations of the trajectories around each of the equilibrium points. Based on the results of Section 8.5, the equilibrium points are saddles for even values of n and stable focuses for odd values of n.

8.6 Linearization Around Equilibrium Points

329

1

0.5

Figure 8.17. The trajectories for (8.21) using the initial conditions given in (8.22). (The dotted lines are in the direction of the eigenvectors corresponding to the Jacobian matrix at xe = (1.5, 0)T .)

x2

0

−0.5

−1

0.5

1

1.5

2

x

1

Using the following initial conditions:         1 1.1 2 1.9 ; (x0 )b = ; (x0 )c = ; (x0 )d = (x0 )a = 1 1 −1 −1 (8.22) we obtain the phase-plane portrait shown in Figure 8.17. In the figure, we see that the eigenvectors found from the linearized models around the saddle are consistent with the actual trajectories (at least locally). Also, for the linearized models around the focus, we can also note that the (1, 2)th element of the Jacobian matrix is positive and thus should have a clockwise rotation around the focus. The plots in the figure show that this is consistent with the nonlinear trajectories as well. We now end this section with an example that the linearization approximation should not be used to approximate the trajectories around non-hyperbolic equilibrium points. EXAMPLE 8.6.

Consider the system     d x2 x1 = x2 −x1 − 6x21 x2 dt

(8.23)

The equilibrium point is at the origin, and the linearized equation around the origin is given by   d 0 1 x= x −1 0 dt The eigenvalues of the Jacobian matrix are ±i, which predicts that the trajectories around the origin should be close to those corresponding to centers. The plot of the trajectory starting at x0 = (−0.15, 0.1)T is shown in Figure 8.18. The trajectory starts out different from trajectories around centers, but at longer times, it does approach a stable focus, with very slow movement toward the origin. It can be shown in the next section that the origin in this example is asymptotically stable (even though it goes very slowly as is nears the origin). Nonetheless, the linearization does predict the clockwise rotation.

2.5

330

Qualitative Analysis of Ordinary Differential Equations

0.1

x20.05 Figure 8.18. The trajectory of (8.23) starting at x0 = (−0.15, 0.1)T . 0

−0.1

−0.05

0

x1

8.7 Method of Lyapunov Functions In the previous section, we noted that the linearization approach is limited to the local analysis of hyperbolic equilibrium points. A more general approach to assess the stability of equilibrium points is an approach known as Lyapunov’s method for stability analysis. We assume that the origin will be the equilibrium point. If this is not the case, we could always translate the axes such that, under the new coordinates, the origin is the equilibrium point. Definition 8.6. Let the origin be an equilibrium point of dx/dt = f (x) and let D be an open neighborhood of the origin. Then for x ∈ D, a scalar function V (x) is a Lyapunov function in D for the dynamic system if V (x) is positive definite, that is, V (x) > 0 for x = 0, V (0) = 0, whereas dV/dt is negative semi-definite, that is, dV dV (x) = f≤0 dt dx Let the origin be an equilibrium point of dx/dt = f (x) and let D be an open neighborhood of the origin. If there exists a Lyapunov function V (x), then the origin is a Lyapunov stable point. If in addition, dV/dt is negative definite, that is, dV/dt < 0 for x = 0, then the origin is asymptotically stable. Likewise, if there exists a positive definite function V (x) such that dV/dt > 0, then the origin is unstable.

THEOREM 8.3.

EXAMPLE 8.7.

Consider the system   d x2 x= −x1 − x21 x2 dt

Let V (x) = 21 xT x, then     dV d = xT x = x1 x2 + x2 −x1 − x21 x2 = −x21 x22 dt dt Thus as long as  ≥ 0, the origin is Lyapunov stable. If  > 0, the origin is asymptotically stable.

8.7 Method of Lyapunov Functions

331

Referring back to Example 8.6, the system there is for the case with  = 60. In that example, we noted that the origin is a non-hyperbolic equilibrium point and that the linearization approach is not applicable. We see that the Lyapunov function approach was still able to assess the stability around the origin. For  = 0, the nonlinear system reduces to a linear system in which the origin will be a center, which is Lyapunov stable but not asymptotically stable.

Although the method of Lyapunov functions is a powerful and general method for the determination of stability of equilibrium points, the biggest problem is the search for these functions. Nonetheless, several candidates are available for one to try. These include the following: 1. V (x) = xT Qx where Q is positive definite. 2. Krasovskii Forms. V (x) = f(x)T Qf(x) 3. Rosenbrock’s Function. V (x) =

N 

|f j (x)|

j =1

One can also use Lyapunov functions to show whether a linear system given by EXAMPLE 8.8.

d x = Ax dt is asymptotically stable without having to determine the eigenvalues. Let P be a symmetric positive definite matrix; then by definition, V = xT Px > 0. Taking the derivative of V , we have       dV d T d = x Px + xT P x = xT AT P + PA x dt dt dt which will be a Lyapunov function if we can find P > 0 such that N = AT P + PA is a negative definite matrix. Thus we can choose any N and try to solve for X in AT X + XA = N

(8.24)

and if X is positive definite, then the linear system will be asymptotically stable. Thus stability of the linear system can be determined by solving the Lyapunov matrix equation (8.24) (which is a special case of Sylvester matrix equation given in (1.23) ) and then proving that the solution is positive definite.6 For instance, let   −2.5 4.5 A= 0.5 −2.5

332

Qualitative Analysis of Ordinary Differential Equations

then with N = −I, we have the solution of (8.24) to be   0.2625 0.3125 X= 0.3125 0.7625 Because X can be shown to be positive definite using Sylvester’s criterion in Theorem A.2, we can conclude that dtd x = Ax will be asymptotically stable. This example may appear to be unnecessary, because one should be able to solve for the eigenvalues of A. Instead, it shows an alternative approach for stability analysis without the need for solving eigenvalues, which may not be easily analyzed if matrix A is large on contains unspecified parameters.7

8.8 Limit Cycles Besides the possibility of having multiple isolated equilibrium points, some nonlinear systems can also produce sustained periodic oscillations called limit cycles. An isolated closed trajectory (cycle) of an autonomous system is called a limit cycle. Limit cycles are different from centers in several ways. One difference is that limit cycles can either be stable, unstable, or partially unstable. Another difference is that centers are trajectories that contain the initial point. However, a stable limit cycle is a periodic trajectory that is approached asymptotically from an initial point, yet do not contain the initial point.

EXAMPLE 8.9.

Consider the van der Pol’s equation

dx1 = x2 dt dx2 = x2 (1 − x21 ) − x1 (8.25) dt The origin is an equilibrium point, and the linearized equation around the origin is given by   d 0 1 x= x −1 1 dt Because the eigenvalue of the Jacobian is given by (0.5 ± 0.866i), the behavior around the origin will be an unstable focus. However, the trajectory does not go unbounded. Instead, it settles down into a limit cycle, as shown in Figure 8.19.

8.8.1 Limit Cycle Existence Theorems There are several methods to assess the existence of limit cycles. For second-order ´ systems, we have two important results: Bendixson’s criterion and the PoincareBendixson theorem. 6

7

The Lyapunov function approach has also been used to prove Routh-Hurwitz stability criterion for a given characteristic polynomial (see, e.g., C. T. Chen, Linear System Theory and Design, Oxford University Press, 1984). This is also the main utility of the Routh-Hurwitz method for stability analysis.

8.8 Limit Cycles 3 2

x2

1 0 -1 -2 -3 -3

-2

-1

0

1

2

3

x1 Figure 8.19. Phase-plane plot of the van der Pol system given in (8.25). THEOREM 8.4.

Bendixson’s Criterion. For the second-order autonomous system given

by d x = f(x) dt suppose that the divergence ∇ ·f =

∂ f1 ∂ f2 + ∂x1 ∂x2

is not identically zero, nor does it change sign in a simply connected open region D; then no limit cycles exist in D. First assume that there is a limit cycle in D and that the limit cycle is contained in a closed curve C. Let S be the closed region enclosed by C. Then using Green’s lemma (cf. Equation 5.1)on the divergence of f in region S, we have    3 ∂ f1 ∂ f2 + dS = f 1 dx2 − f 2 dx1 ∂x2 S ∂x1 C   T dx2 dx1 f1 = − f2 dt dt dt 0

PROOF.

=

0

where T is the period of the cycle. However, for the surface integral to be zero, we need the divergence of f to either be zero or change sign.

Consider the van der Pol system given in (8.25). Calculating the divergence of f, we have

EXAMPLE 8.10.

∇ · f = 1 − x21 Let D be a region bounded by a circle centered at the origin having a radius, r < 1. The divergence of f in D is always positive. We then conclude that in this region, there are no limit cycles. This can be verified by observing the plot given in Figure 8.19. However, note that the criterion does not prevent portions of limit cycles to be in the region that satisfy the conditions of the criterion.

333

334

Qualitative Analysis of Ordinary Differential Equations

n

Cout

f

´ Figure 8.20. The conditions given in Poincare-Bendixson’s theorem.

M Cin ´ Poincare-Bendixson’s Theorem. Let M be a region bounded by two nonintersecting closed curves Cout and Cin , where Cin is inside Cout , as shown in Figure 8.20. For the second-order autonomous system given by

THEOREM 8.5.

d x = f(x) dt If 1. There are no equilibrium points inside M. 2. Along Cin and Cout f·n≤0 where n is the outward unit normal vector of region M, 3. Inside M   −x2 = 0 (f)T x1

(8.26)

(8.27)

then a stable limit cycle exists in M. ´ This statement of the Poincare-Bendixson’s theorem gives a sufficient condition for the existence of a limit cycle in M. The condition in (8.27) means that the trajectories inside M never point in a radial (in or out) direction.8 The theorem, however, does not identify whether there is only one or a multiple number of limit cycles in M. Furthermore, this result cannot be extended to third- or higher order autonomous nonlinear systems.

EXAMPLE 8.11.

Consider the following autonomous system:          x1 d x2 x1 = + 1 − x21 + x22 x2 x1 x2 dt

(8.28)

Next, choose Cin to be a circle centered around the origin of radius rin , and choose Cout to be a circle centered around the origin of radius rout , where 0 < rin < rout . The outward unit normal vectors at Cin are   y − cos (θ) where θ = tan−1 nin = , x2 + y2 = rin − sin (θ) x 8

´ A more general version of the Poincare-Bendixson’s theorem simply states that if a trajectory never leaves the region M, then either the trajectory is a limit cycle or approaches a limit cycle in M.

8.8 Limit Cycles

335 2 1.5 1

Figure 8.21. A plot of some trajectories of (8.28) showing that the unit circle is a limit cycle.

x2

0.5 0

-0.5 -1 -1.5 -2 -2

-1

0

1

x1

whereas the outward unit normal vectors at Cout are   y cos (θ) where θ = tan−1 nout = , x2 + y2 = rout sin (θ) x Along Cin , we find that

  2 sin(θ)2 f · n = −rin 1 − rin

0 ≤ θ ≤ 2π

whereas along Cout ,

  2 f · n = rout 1 − rout sin(θ)2

0 ≤ θ ≤ 2π

and we can see that we need rin < 1 and rout > 1 to satisfy (8.26). As for (8.27), we have     −x2 = − x21 + x22 = 0 for (x1 , x2 ) ∈ M fT x1 A plot of some trajectories of the system is shown in Figure 8.21. From the figure, ´ one can observe that the Poincare-Bendixson theorem would have predicted a limit cycle inside the annulus region between the outer circle of radius greater than 1 and an inner circle of radius less than 1.

` 8.8.2 Poincare´ Maps and Lienard Systems ` The Poincare-Bendixson theorem is a tool to determine the existence of a stable limit cycle inside a region in a phase-plane, that is, a second-order system. In higher order cases, the theorem may no longer hold because other trajectories may be present, such as quasi-periodic or chaotic trajectories. To determine the presence of limit cycles in these cases, a tool known as Poincare´ maps (also known as firstreturn maps) can be used. It can also be used to show uniqueness of stable limit cycles. Let S be an (n − 1)-dimensional hypersurface that is transverse to the flow of an n-dimensional autonomous system dx/dt = f(x) (i.e., none of the trajectories will be parallel to S). A Poincare´ map P is a mapping obtained by following the trajectory

2

336

Qualitative Analysis of Ordinary Differential Equations

S

xk+1 xk

Figure 8.22. Poincare´ map based on surface S.

of an intersecting point x∗k (of the trajectory with S) to the next intersecting point x∗k+1 , that is, x∗k+1 = P (x∗k )

(8.29)

This is shown diagramatically in Figure 8.22. Note that the hypersurface S is often a bounded or semibounded region in n − 1 dimension that slices the n dimensional region containing the limit cycle. If a point x∗eq in the surface S is the fixed point of the Poincare´ map, that is,   x∗eq = P x∗eq , then x∗eq belongs to a limit cycle.9 A discrete version of Lyapunov functions can also be used to determine the stability of the limit cycle. This principle will be used in the following example to show the uniqueness of a limit cycle of a ` particular class of second-order equations known as Lienard systems.

EXAMPLE 8.12.

` Consider the second-order system known as the Lienard system

given by d2 x dx + f (x) + g(x) = 0 (8.30) 2 dt dt ` Instead of the usual state-space representation, the Lienard coordinates can be used by defining y as  x dx y= + F (x) where F (x) = f (ξ)dξ (8.31) dt 0 Then (8.30) becomes d dt



x y



 =

y − F (x) −g(x)

 (8.32)

Now assume the following conditions: 1. The function g(x) is an odd function with g(x) > 0 if x > 0. 2. The integral F (x) is also an odd function and has three roots given by x = 0, x = q > 0, and x = −q < 0, such that f (0) < 0 and f (x) > 0 at x = ±q. 3. F (x) → ∞ as x → ∞.

9

We are using a technical distinction between fixed points of discrete maps such as Poincare´ maps and equilibrium points for continuous maps that apply to differential equations.

8.8 Limit Cycles

337

y

y=F(x)

S

+

Y

` Figure 8.23. The nullclines of the Lienard system. q

-q

Y+

S-

` If all these conditions are satisfied, then the Lienard system given by (8.30) will have exactly one stable limit cycle, as we show next. (When we refer to the ` Lienard system that follows, we are assuming that these conditions are already ` attached.) The relevance of Lienard systems is that it is a class of autonomous oscillators of which the van der Pol equation is a special case. The Jacobian matrix resulting from linearization of (8.32) at the origin is given by  J0 =

−f (x) −dg/dx

1 0

   

(x,y)=(0,0)

Because tr(J 0 ) = −f (0) > 0 and det(J 0 ) = dg/dx > 0, the origin is hyperbolic and is either an unstable node or unstable focus (see Figure 8.16). Let S+ be the strictly positive y-axis (i.e., excluding the origin); then S+ is a nullcline where the trajectories are horizontal and pointing to the right ( dy/dt|(x=0) = g(0) = 0 and dx/dt|(x=0) = y > 0). Another nullcline S− is the strictly negative y-axis where the trajectories are horizontal and pointing to the left. The other two nullclines Y − and Y + will be the graph y = F (x) with x = 0. The trajectories at Y − are vertical and pointing down when x > 0, whereas the trajectories at Y + are vertical pointing up when x < 0 ( dy/dt = −g(x) < 0 for x > 0 and dy/dt = −g(x) > 0 when x < 0). The nullclines are shown in Figure 8.23. Using all the nullclines, one can apply Poincare´ -Bendixson’s theorem to prove that a limit cycle exists in a region M whose outer boundary has a distance from the origin that is greater than q (the root of F (x)) and inner boundary is a circle of a small radius  > 0 surrounding the origin. However, this does not prove the uniqueness of the limit cycle. To do so, we need to find a unique fixed point of a Poincare´ map, which we will choose S+ to help define the Poincare´ map P. Because we are only given conditions for g(x) and F (x), we cannot evaluate the actual Poincare´ map P.10 Instead, we use a Lyapunov function defined by  x 1 2 V (x, y) = y + g(ξ)dξ 2 0 whose derivative is given by dV dy dx = y + g(x) = −g(x)F (x) dt dt dt 10

` Even if we are given f (x) and g(x), the Lienard equation is an Abel equation, which can be very difficult to solve in most cases.

-

x

338

Qualitative Analysis of Ordinary Differential Equations

` Furthermore, note that because F (x) and g(x) are odd functions, the Lienard system is symmetric with respect to both x and y; that is, the same equation (8.32) results after replacing x and y by & x = −x and & y = −y, respectively. Thus instead of the full Poincare´ map P, we can just analyze the map & P, which is the map of the trajectory starting at a point in S+ to a first intersection in S− , and conclude P(y∗ ) due to the special symmethat y∗ is in a limit cycle if and only if y∗ = −& ∗ + try. The fixed point (0, y ) at S is unique if the Lyapunov function between Poincare´ maps at S+ is equal, that is,   V (0, y∗ ) = V (0, P(y∗ )) or V (0, y∗ ) = V 0, −& P(y∗ )   Let Vy0 = V 0, −& P(y0 ) − V (0, y0 ). Starting at (0, y0 ) with y0 > 0, the trajectory will intersect the nullcline Y − . If the intersection with Y − is at x ≤ q, then dV/dt > 0, which means Vy0 > 0. However, if the intersection with Y − is at x > q, we can show that the difference Vy0 > 0 decreases monotonically with increasing y0 , as sketched here: 1. The value of Vy0 can be split into three paths yielding  t1  t2 

Vy0 = V (x(t), y(t))dt + V (x(t), y(t))dt + 0

t1

t3

V (x(t), y(t))dt t2

where t1 is the first instance that x = q, t2 is the second instance that x = q as the trajectory loops back toward the negative y-axis, and t3 is the time when x = 0 and y = & P(y0 ) at the negative y-axis. 2. The first and third integrals can be evaluated as  q  t1  t3  0 F (x)g(x) F (x)g(x) Vdt = − dx and Vdt = − dx 0 0 y − F (x) t2 q y − F (x) and will both decrease for an increase in y0 because the denominator will be a more positive value for the integral from 0 to t1 , and the denominator will be a more negative value for the integral from t2 to t3 . 3. For the second integral  y− (q)  t2 Vdt = F (x(y))dy t1

y+ (q)

the arc from y+ (q) to y− (q) increases in size to the right and F (x) > 0 along this arc as y0 is increased. However, because y− (q) < 0 < y+ (q), this integral will also decrease as y0 increases. 4. Because all three integrals decrease as y0 increases, we conclude that Vy0 decreases monotonically as y0 is increased. Following the same arguments of the integrals, it can be shown that Vy0 → −∞ as y0 → ∞. Combining these results, we can see from Figure 8.24 that there is only one fixed point in S+ (equiv. in S− ); that is, the limit cycle of the ` Lienard system is stable and unique. Because the Poincare´ map is a discrete map xk+1 = P(xk ), one could test the stability of the limit cycle passing through a fixed point x∗ in the surface S by introducing a perturbation k , that is, x∗ + k+1 = P (x∗ + k )

8.8 Limit Cycles

339

Vy >0

}

0

}

0

Vy

Figure 8.24. The change in Lyapunov function Vy0 as a function initial point y0 , showing only one unique fixed point y∗ of the Poincare´ map based on S+ of the ` Lienard system.

y0

0

y*

Using Taylor series expansion and truncating the higher order terms, we end up with a linearized Poincare´ map J P for the perturbation k+1 = J P k where  dP  JP = dx x=x∗

(8.33)

Using Theorem 7.2, the perturbations will die out if all the eigenvalues λi , i = 1, . . . , n, of the linearized Poincare´ map J P have magnitudes less than 1. These eigenvalues are also known as the Floquet multipliers of the Poincare´ map P. However, similar to the non-hyperbolic case of an equilibrium point in the continuous case, the stability cannot be determined using linearization if any of the eigenvalues of J P have a magnitude equal to 1. In those cases, the Lyapunov analysis or other nonlinear analysis methods need to be used.

8.8.3 Nonlinear Centers If the equilibrium points are non-hyperbolic points, there are still cases in which the surrounding neighborhood will have periodic trajectories (or orbits). If this is the case, the equilibrium point is known as a nonlinear center. However, periodic orbits surrounding a nonlinear center are not the same as limit cycles because they are not isolated; that is, any small perturbation will immediately lie in a neighboring periodic orbit. Two special cases will yield a nonlinear center. The first case involves the conservative system, when a Lyapunov function V (x) can be found around the equilibrium point xeq such that the Lyapunov functions are constant around the neighborhood of the equilibrium point. One class of second-order conservative system is given in Exercise E8.15. The other case is when the system belongs to a class of reversible systems, where the trajectories are symmetric with respect to time and one of the coordinates, that is, with τ = −t and & y = −y d dt



 x y



 =

f 1 (x, y) f 2 (x, y)

→

d dτ



 x & y



 =

f 1 (x,& y) f 2 (x,& y)

340

Qualitative Analysis of Ordinary Differential Equations

0.6

0.4

0.2

y

Figure 8.25. Plot of trajectories for (8.34) with initial points at (1, 0), (0.75, 0), (0.5, 0), (0.25, 0), and (0.11, 0).

0

−0.2

−0.4

−0.6

−0.5

0

0.5

1

x

EXAMPLE 8.13.

Consider the reversible system       d y 1 − y2 x = dt y −x + y2

(8.34)

The origin is an equilibrium point that is non-hyperbolic, with the Jacobian matrix at the origin given by   0 1 J (x = 0) = −1 0 Various trajectories are shown in Figure 8.25, starting at different initial conditions. Note that the shapes of the periodic trajectories are not circular nor elliptic the farther they are from the origin. Furthermore, the rotation is clockwise, which can be predicted from J 12 > 0 (cf. Section 8.5).

8.9 Bifurcation Analysis The qualitative analysis discussed thus far has mostly dealt with fixed coefficients. As can be anticipated, if some of the system coefficients are replaced by parameters, the behavior will most likely change. It could mean the addition or loss of equilibrium points or limit cycles, as well as changes in the stability of these behaviors. A brief discussion on bifurcation analysis is also included in Section H.1 as an appendix. These analysis are aimed at how the phase portraits (i.e., collective behavior of the system) are affected by variations in critical parameters. 8.10 EXERCISES

E8.1. Consider a scalar system involving a parameter 0 < q < 1 given by + dx 0 for x ≤ 0 = f(x) = q for x > 0 x dt with initial condition x(0) = 0. Show that the following function is a solution: + 0 for t ≤ τ x(t) = for t > τ [(t − τ) (1 − q)]1/(1−q)

8.10 Exercises

for all τ > 0. This means that the solution is not unique, and this is possible because although f (x) is continuous, its derivative df/dx is not continuous at x = 0. E8.2. Consider the following autonomous system, with a = 0, dx = −ay dt dy = az dt dz = −a2 z dt Show that the equilibrium point is Lyapunov stable but not quasiasymptotically stable. E8.3. Consider the following system given in Glendinning (1994):   dx xy = x − y + x x2 + y2 + ! dt x2 + y2   dy x2 = x + y − y x2 + y2 − ! dt x2 + y2 1. Determine the equilibrium points. 2. Obtain a direction field for this system in the domain −2 ≤ x ≤ 2 and −2 ≤ y ≤ 2. (We suggest a 20 × 20 grid.) Ascertain from the direction field whether the equilibrium is Lyapunov stable, quasi-asymptotic stable, or asymptotically stable. 3. Use the initial value solvers and obtain the trajectories around the equilibrium. E8.4. For the linear second-order system dx/dt = Ax, once it can be determined to be a focus or center, show that the direction of rotation can be determined to be clockwise or counterclockwise depending on whether a12 > 0 or a12 < 0. One approach is given by the following steps: 1. Show that a necessary condition for focus or centers is that a12 a21 < 0. 2. Let r = x1 δx + x2 δy be the position vector of x and v = y1 δx + y2 δy be the tangent to trajectory at x, where y = Ax. Show that the cross product is given by r ⊗ v = ξδz where ξ = a21 x21 + (a22 − a11 ) x1 x2 − a12 x22 3. Using the fact that a12 a21 < 0, show that if a21 > 0, we must have a12 < 0 then ' (2 a22 − a11 1 ξ> x1 − a12 x2 ≥ 0 −a12 2 and the rotation becomes counterclockwise. Next, you can show the reverse for clockwise rotation. E8.5. Show that the conditions given in (8.13) will indeed determine whether the improper node is either S-type or Z-type. (Hint: To obtain an S-type improper node, trajectories at either side of v will have to be counterclockwise for

341

342

Qualitative Analysis of Ordinary Differential Equations

stable equilibrium points and clockwise for unstable equilibrium points. For the Z-type improper nodes, the situations are reversed.) E8.6. Obtain the trajectories in the phase plane for the degenerate cases with A = 0 where both eigenvalues are zero. E8.7. For each of the cases for A that follow, determine the type of equilibrium points and do a phase-plane analysis of each case for dx = Ax dt by plotting trajectories at various initial points surrounding the equilibrium point. Also include supplementary information such as bounding ellipse, direction of rotation, shapes, straight line trajectories, and so forth appropriate for the type of equilibrium points.  a) A =  d) A =  g) A =

1 4

2 1



 b) A = 

1 −4

2 1

−2 3

1 −2

 e) A = 

 h) A =

−1 −2

2 1

0 −6

1 5

2 4

1 2



 c) A =



 f) A =



 i) A =

2 −3

4 −2

−5 0

0 −5

0 0

−3 0







E8.8. Let x1 and x2 be the population of two competitive species 1 and 2. One model for the population dynamics is given by    d x1 0 b − Ax x= 0 x2 dt where b and A are constant, with A nonsingular. 1. Show that there are four equilibrium points given by ⎞ ⎛ ⎛ ⎞ ⎛ b1 0 ⎟ ⎜ ⎜ ⎟ ⎜ 0 ⎟ ; (xe ) = ⎜ a11 ⎟ ; (xe ) = ⎜ (xe )1 = ⎜ 2 3 ⎠ ⎝ ⎝ ⎠ ⎝ b2 0 0 a22

⎞ ⎟ ⎟ ; (xe ) = A−1 b 4 ⎠

2. Prove that if b is a positive vector and A is a symmetric, positive definite, strictly diagonally dominant matrix with a12 = a21 ≤ 0, then all the equilibrium points will be hyperbolic, (xe )2 and (xe )3 are saddles, (xe )1 is an unstable node (or star), and (xe )4 is a stable equilibrium point. (Hint: For (xe )4 , one can use Gershogorin circles.) 3. Consider the specific case:     2 −1 2 A= and b = −1 1.5 3 Predict the local trajectories around the equilibrium points. Obtain a rough sketch of the phase portrait, and then verify using a direction field and using ODE IVP solvers.

8.10 Exercises

E8.9. Consider the following coupled equations describing the dynamics of a nonisothermal continuously stirred tank reactor based on energy and mass balance for a first-order reaction A → B: dC F = (Cin − C) − k0 Ce−E/(R(T +460)) dt V dT F UA (− H) = k0 Ce−E/(R(T +460)) − (T − T j ) (T in − T ) + dt V ρc p Vρc p where C and T are the concentration of A and temperature in the reactor, respectively. Using the following parameter set: F : V: T in : Cin

H : ρc p : U: A: ko : E: R: Tj:

volumetric flow rate liquid volume temperature of feed : concentration of A in feed molar heat of reaction heat capacity per volume heat transfer coefficient area for heat transfer Arrhenius frequency factor activation energy universal gas constant jacket temperature

= 3,000 ft3 /hr = 750 ft3 = 60 o F = 0.132 lbmol/ft3 = −45,000 BTU/lbmol = 33.2 BTU/( ft3 o F ) = 75 Btu/(hrft2 o F ) = 1221 ft2 = 15 ×1012 hr−1 = 32,400 Btu/(lbmol o F ) = 1.987 Btu/(lbmol o R) = 55 o F

1. Find all the equilibrium points and show that they are all hyperbolic. 2. Based on linearization, determine the local behavior of the trajectories surrounding the equilibrium points. 3. Plot a direction field in a phase that contains all three equilibrium points. Then using an ODE IVP solver, obtain the phase portrait that shows the behavior of trajectories using various initial conditions. E8.10. For the second-order system described by   d h(x) k x= x −k h(x) dt where h(x) is a scalar function of x1 and x2 and k is a real-valued constant. 1. Show that the origin is a non-hyperbolic equilibrium point if h(0) = 0. 2. Show that if h(x) is negative definite (semi-definite), then the origin is asymptotically stable (Lyapunov stable) by using the Lyapunov function V (x) = xT x. 3. Determine the system stability/instability if h(x) is positive definite. E8.11. Use the Lyapunov function approach as in Example 8.8 to show that the following linear equation is stable:   d −5 1 x= x 2 −4 dt E8.12. Use Bendixon’s criterion to show that the system given in Example 8.5,     d x2 x1 = x2 cos (πx1 ) − x2 dt will never have limit cycles.

343

344

Qualitative Analysis of Ordinary Differential Equations

E8.13. For the system given by   d x2 x = f(x) = −x1 − x2 g(r) dt " where r = x21 + x22 . 1. Let g(r) be any monotonic function in the range r ≥ 0, with a root at ´ r = r∗ > 0, that is, g(r) < 0 for r < r∗ and g(r) > 0 for r > r∗ . Use PoincareBendixson’s theorem to show that a limit cycle will exist around the origin. Furthermore, show that the limit cycle will have a circular shape. (Hint: Apply the theorem to the annular region between a circle having a radius less than r∗ and a circle of radius greater than r∗ .) Test this for g(r) = (r − 1)(r + 1). ´ 2. Let g(r) = cos(2πr/R) for some fixed R. Using Poincare-Bendixson’s theorem, show that there will be multiple circular limit cycles having radius r∗ at the roots of g(r), and determine which are stable limit cycles and which are unstable limit cycles. E8.14. Consider the following system   x2 d     x = f(x) = 2 − 21 − 1 − x21 e−x1 x2 − x1 dt ` Check that the conditions of the Lienard system given in Example 8.12 are satisfied and thus this equation will have a stable limit cycle. Solve the system using the IVP solvers at different initial conditions to show that the limit cycle is stable and unique. E8.15. Consider a second-order process given by x¨ = h(x) or, in state-space representation,     d x y = y h(x) dt 1. Let xr be any root of h(x) such that   d h(x) dx

0. 1. From each equation we obtain two graphs: y = g 1 (x) = ax and y = g 2 (x) = x2 /[b(1 + x2 )]. The equilibrium points are at the intersection of both graphs. Plot both graphs under a fixed b to show that with alo > 0 there is a range of alo < a < ahi such that the number of equilibrium points will change from three to two. Also, for a > ahi , there will only be one equilibrium point. 2. Determine the stability of the equilibrium points. 3. Draw a bifurcation diagram for this system for b = 1, and classify the type of bifurcations at the bifurcation points. E8.19. The Brusselator reaction is given by following set of reactions: A

→

x

B+x

→

y+C

2x + y

→

3x

x

→

D

where the concentrations of CA, CB, CC, and CD are assumed to be constant. Furthermore, with the reactions assumed isothermal,the specific kinetic rate

345

346

Qualitative Analysis of Ordinary Differential Equations

coefficients ki for the ith reaction will be constant. The rate of change in the concentrations of x and y will then be given by dCx = k1 CA − k2 CBCx + k3 C2x Cy − k4 Cx dt dCy = k2 CBCx − k3 C2x Cy dt 1. Obtain the equilibrium point of the reaction system. 2. Choosing CB as the bifurcation parameter, find the critical point CB,h under which the Brusselator can satisfy the conditions for Hopf bifurcation. 3. Using the IVP solvers to simulate the process, determine whether the Brusselator exhibits supercritical or subcritical Hopf bifurcation under the following fixed parameters: k1 = 1.8, k2 = 2.1, k3 = 1.2, k4 = 0.75, and CA = 0.9.

9

Series Solutions of Linear Ordinary Differential Equations

In this chapter, we focus our attention on obtaining analytical solutions of linear differential equations with coefficients that are not constant. These solutions are not as simple as those for which the coefficients were constant. One general approach is to use a power series formulation. In Section 9.1, we describe the main approaches of power series solution. Depending on the equation, one can choose to expand the solution around an ordinary point or a singular point. Each of these choices will determine the structure of the series. For an ordinary point, the expansion is simply a Taylor series, whereas for a singular point, we need a series known as a Frobenius series. Although the power series method is straightforward, power series solutions can be quite lengthy and complicated. Nonetheless, for certain equations, solutions can be found based on the parameters of the equations, thus yielding direct solutions. This is the case for two important classes of second-order equations that have several applications. These are the Legendre equations and Bessel equations, which we describe in Sections 9.2 and 9.3, respectively. We have also included other important equations in the exercises, such as hypergeometric equations, Jacobi equations, Laguerre equations, Hermite equations, and so forth, where the same techniques given in this chapter can be used to generate the useful functions and polynomials. Fortunately, the special functions and polynomials that solve these equations, including Legendre polynomials, Legendre functions and Bessel functions, are included in several computer software programs such as MATLAB.

9.1 Power Series Solutions Consider the first-order equation dy − ay = 0 with y(0) = y0 dx

(9.1)

We can apply the successive substitution approach known as Picard’s method and obtain a series solution. The method begins with an integration of (9.1),  x y(x) = y0 + a y(τ1 )dτ1 0

347

348

Series Solutions of Linear Ordinary Differential Equations

and continued with recursive integrations on y(τi ), i = 1, 2, . . .,   x  τ1 y0 + a y(x) = y0 + a y(τ2 )dτ2 dτ1 0

=



y0 + axy0 + a

x



0 τ1

2 0



 y0 + a

0



τ2

y(τ3 )dτ3 dτ2 dτ1 0

=

  (ax)2 (ax)3 y0 1 + ax + + + ··· 2! 3!

=

y0 eax

This example suggests that some of the solutions to linear differential equations could be represented by power series. Furthermore, it shows that, as in the preceding process toward forming eax , the series solutions themselves can often generate the construction of new functions. For instance, we can show that Legendre polynomials and Bessel functions could be defined from series solutions of certain classes of second-order linear differential equations. However, instead of using Picard’s method, we approach the solution by first constructing a power series that contains unknown parameters and coefficients. The parameters are then evaluated after substitution of the power series into the differential equations. Consider the homogeneous linear Nth -order differential equation ψN (x)

dN y dy + · · · + ψ1 (x) + ψ0 (x)y = 0 N dx dx

(9.2)

Definition 9.1. A point x∗ is an ordinary point of (9.2) if ψN (x∗ ) = 0 and ψ j (x∗ ) < ∞, j = 0, 1, 2, . . . , (N − 1). A point x∗ is a singular point of (9.2) if ψN (x∗ ) = 0 or ψ j (x∗ ) = ±∞ for some 0 ≤ j < N. Furthermore, if x∗ is a singular point, then it is known as a regular singular point if    ψ j (x)  0 and is related to the Gamma function by (x)Pn =

(x + n) (x)

(9.11)

7. The generalized hypergeometric series, denoted p F q , is an infinite series defined as follows: p F q (a1 , . . . , a p ; b1 , . . . , bq ; x) =

∞  (a1 )Pn · · · (a p )Pn xn (b1 )Pn · · · (bq )Pn n!

(9.12)

n=0

where (a1 )Pn , (a2 )Pn and so forth are Pochhammer symbols defined in (9.10) Special cases includes Gauss’ hypergeometric series defined as ∞  (a)Pn (b)Pn xn 2 F 1 (a, b; c; x) = n! (c)Pn n=0

(9.13)

9.1 Power Series Solutions

351

1

0.5

Figure 9.2. A plot of the error function, erf(x).

erf(x)

0

−0.5

−1 −2

−1

0

1

x

and the confluent hypergeometric series defined as ∞  (a)Pn xn 1 F 1 (a; b; x) = (b)Pn n!

(9.14)

n=0

These functions are useful in evaluating several series solutions. They were originally used to solve differential equations known as hypergeometric equations (see Exercise E9.3). The MATLAB command for y=

p F q (a1 , . . . , a p ; b1 , . . . , bq , x)

is y=hypergeom([a1,...,ap],[b1,...,bp],x). 8. The error function, denoted by erf(x), is defined as 2 erf(x) = √ π



x

e−y dy 2

(9.15)

0

A plot of the error function is shown in Figure 9.2. In the limit as x approaches infinity, we have lim erf(x) = 1

x→∞

An associated function known as the complementary error function, denoted by erfc(x), is defined as 2 erfc(x) = 1 − erf(x) = √ π



∞

e−y dy 2

(9.16)

x

In MATLAB, the function erf is available for evaluating the error function. In the sections that follow, we first tackle the series solutions expanded around an ordinary point. Then we follow it with the discussion of the series solutions expanded around a regular singular point. We initially explore the solutions of highorder equations. However, at some point, we must limit it to second-order differential equations to allow for some tractable results.

2

352

Series Solutions of Linear Ordinary Differential Equations

9.1.1 Series Solutions Around Ordinary Points Consider the homogenous Nth -order linear differential equation given by N 

p j (x)

j =0

dj y =0 dx j

(9.17)

then x = 0 is an ordinary point of (9.17) if the coefficient functions p j (x) are all analytic around x = 0; that is, the coefficient functions can be expanded as p j (x) =

∞ 

ρ j,n xn

j = 0, 1, 2, . . . , N

(9.18)

n=0

with the additional condition that ρN,0 = 0. Let x = 0 be an ordinary point of the Nth -order linear differential equation described by (9.17) and (9.18). The general series solution is then given by

THEOREM 9.1.

y=

∞ 

an xn

(9.19)

n=0

with N arbitrary integration constants a0 , a1 , . . . , aN−1 , and the remaining constants an+N , n = 0, 1, . . ., satisfy the following recursion formulas: an+N =

n+N−1 

φn,k ak

n = 0, 1, . . . , ∞

(9.20)

k=0

where ρ0,n−k +

N 

ρ j,n−k+j

j =1

φn,k = (−1)

ρN,0

j −1 

(k − i)

i=0 N 

(9.21)

(n + i)

i=1

with ρ j, = 0 PROOF.

 2. However, in general, the determination of the symmetries of a differential equation can be a very long and difficult process. One type of transformation is easy to check for symmetry. These are the similarity transformations (also known as scaling or stretch transformations), given by & u = λ−γ u and

x&k = λ−αk xk for k = 1, . . . , n

(11.84)

where λ is known as the similarity transformation parameter, and at least two of the exponents αk must be nonzero. To determine whether a given partial differential equation admits a similarity transformation, one would only need to substitute the transformations given in (11.84) into the given differential equation (11.81) and determine whether there exists values of αk and γ that would yield an equation that does not involve the parameter λ. t, Applying the similarity group of transformations (11.84), t = λα& γ x and u = λ & u to the differential equation given by x=λ&

EXAMPLE 11.13. β

∂u ∂2u +x 2 =A ∂t ∂x where A is a constant, we obtain  γ−α  ∂& u  γ−β  ∂ 2& u λ + λ & x 2 =A ∂& t ∂& x Symmetry is achieved if we set α = β = γ = 1. Conversely, one can show that the following differential equation does not admit symmetry based on a similarity transformation: ∂u ∂ 2 u + 2 = x2 + A ∂t ∂x At this point, we limit the similarity method to handle only partial differential equations for u that depend on two independent variables x and t. For this case, we have the following theorem:

11.5 Similarity Transformations THEOREM 11.3.

441

The partial differential equation   ∂u ∂u ∂ 2 u ∂ 2 u ∂ 2 u F x, t, u, , , 2 , , ,... = 0 ∂t ∂x ∂x ∂x∂t ∂t2

(11.85)

which admits symmetry for the similarity transformations given by& t = λ−α t,& x = λ−β x and & u = λ−γ u, can be reduced to an ordinary differential equation with ζ and ψ as the independent and dependent variables, respectively, where ζ= PROOF.

& xα xα = & tβ tβ

ψ=

and

& uα uα = & tγ tγ

(See Section K.2.3 for proof.)

Theorem 11.3 guarantees that a partial differential equation involving two independent variables that admits a similarity transformation can be reduced to an ordinary differential equation. However, one will need to consider additional complexities. In some cases, especially for linear partial differential equations, there can be more than one set of similarity transformations. One will need to determine whether symmetry applies to the initial and boundary conditions as well. Finally, the ordinary differential equations may not necessarily be easy to solve. In some cases, numerical methods might be needed.

EXAMPLE 11.14.

Consider the diffusion equation given by

∂u ∂2u = μ2 2 (11.86) ∂t ∂x subject to the conditions: u(x, 0) = ui and u(0, t) = u0 . After substitution of & x = λ−β x and & u = λ−γ u, we have t = λ−α t, & λγ−α

∂& u ∂ 2& u = μ2 λγ−2β 2 ∂& t ∂& x

  subject to λ−γ u λ−β x, 0 = ui and λ−γ u (0, λ−α t) = u0 . For symmetry, we need α = 2β and γ = 0, so we √ can set√α = 1, β = 1/2 and γ = 0, yielding the invariants x/ & ψ = u =& u and ζ = x/ t = & t. Substituting these into the partial differential equation (11.86) will yield μ2 The general solution is given by

d2 u ζ du =− 2 dζ 2 dζ



u (ζ) − u (0) = C1 0

ζ



−ζ2 exp 4μ2





ζ dζ = A erf 2μ



√ where A = C1 μ π is an integration constant and erf(x) is the error function defined by  z 2 2 erf(z) = √ e−q dq π 0

442

Linear Partial Differential Equations

1

t=0 t=1

0

0

(u−u )/(ui−u )

0.8

t=2

0.6

t=3 0.4

Figure 11.10. A!plot of (u − u0 )/(ui − u0 ) distribution along x/ 4μ2 t at different instants of t.

t=4

t=5

0.2

0

−2

0

2

4

6

8

10

2 −1/2

(4μ t)

x

with erf(0) = 0 and erf(∞) = 1. After implementing the initial and boundary conditions, that is, u(0, t) = u(ζ = 0) = u0 and u(x, 0) = u(ζ = ∞) = ui , the solution is then given by   x u (x, t) = (ui − u0 ) erf ! + u0 4μ2 t ! A plot of the ratio (u − u0 )/(ui − u0 ) along the normalized distance x/ 4μ2 t at different values of t is shown in Figure 11.10.

For the diffusion equation given in (11.86), suppose the conditions are now given by u(x, 0) = 0, limx→∞ u(x, t) = 0, and ∂u/∂x(0, t) = H. x = λ−β x and & u = λ−γ u, the same partial differAfter substitution of & t = λ−α t, & ential equation results as in Example 11.14, except that the initial and boundary conditions are now given by   u λβ& x, 0 = 0 λγ& ∂& u   β and λγ−β t) = H (0, λα& γ α ∂& x t = 0 u λ& x, λ & limλβ&x→∞ λ &

EXAMPLE 11.15.

For symmetry, we need α = 2β and γ = β, so we √ can set √ α = 1, β = 1/2√and γ= ψ = u/ t = & u/ & t and ζ = x/ t = √ 1/2, yielding the following invariants √ & & x/ t. Substituting these together with u = t ψ(ζ) into the partial differential equation (11.86) yields the following ordinary differential equation: d2 ψ ζ dψ 1 + − ψ =0 dζ2 2μ2 dζ 2μ2 dψ (0) = H. The solution of this equation can be ζ→∞ dζ found in Example 9.2. After applying the boundary conditions, we have     7 2Hμ ζ π −ζ2 /(4μ2 ) ψ = Hζ − √ + e ζ erf ! 4μ2 π 4μ2

subject to lim ψ(ζ) = 0 and

11.6 Exercises

443

3

2

τ = 0.1

1

0

Figure 11.11. A plot of u/H distribution along x ≥ 0 at different instants of τ = 4μ2 t.

u/H

−1

−2

−3

−4

0

τ = 1.0 τ = 5.0 τ = 10. τ = 20

2

4

6

8

x

or in terms of the original variables, ⎡ ⎛ 8   6 92 ⎞⎤ 2 x 4μ t x ⎠⎦ u(x, t) = H ⎣x erfc ! − exp ⎝− ! 2 π 4μ t 4μ2 t where erfc(z) = 1 − erf(z) is known as the complementary error function. A plot showing u/H as a function of x at different values of τ = 4μ2 t is shown in Figure 11.11.

Several textbooks consider the similarity transform methods to be limited to a few applications, citing that only special boundary conditions can match the solutions of the resulting ordinary differential equations. However, this should not preclude the fact that symmetry methods, of which similarity transforms is just one possibility, is actually quite general, especially because of its effectiveness in solving several nonlinear partial differential equations. A more serious practical issue is the complexity of the calculations involved in both finding the symmetries and solving the resulting ordinary differential equations. However, with the availability of improved symbolic equation solvers, the symmetry approaches have become more tractable, and they are gaining more acceptance as a viable approach. 11.6 EXERCISES

E11.1. Obtain the general solutions for the following reducible or reduced homogeneous differential equations: ∂2u ∂2u ∂2u ∂u ∂u + −2 2 +8 +3 + 2u = 0 2 ∂x ∂x∂y ∂y ∂x ∂y   ∂ ∂ Hint: One of the factors of L is 2 − +2 . ∂x ∂y E11.2. For the second-order bivariate hyperbolic equation given by 6

∂2u ∂u ∂u + a(x, y) + b(x, y) + c(x, y)u = 0 ∂x∂y ∂x ∂y

(11.87)

10

444

Linear Partial Differential Equations

1. Define two linear operators: ∂ ∂ Lx = + b(x, y) and Ly = + a(x, y) ∂x ∂y Show that (11.87) can be rearranged in two forms: Lx Ly (u) = ha (x, y)u

or

Ly Lx (u) = hb(x, y)u

where

∂a + a(x, y)b(x, y) − c(x, y) ∂x ∂b hb(x, y) = + a(x, y)b(x, y) − c(x, y) ∂y The functions ha and hb are known as the Laplace invariants of (11.87). 2. If ha = hb = 0, (11.87) will be reducible. Solve the case where a(x, y) = x, b(x, y) = y and c(x, y) = 1 + xy. 3. If only one of the Laplace invariants is zero, one can still proceed by integrating with respect to one independent variable, followed by the integration with respect to the other independent variable. For instance, if Lx Ly u = 0, then first solve Lx z = 0 followed by solving Ly u = z. Using this approach, obtain the general solution of (11.87) for the case where a(x, y) = y, b(x, y) = xy and c(x, y) = xy2 . ha (x, y)

=

E11.3. For the following initial value problem ∂2u ∂2u ∂2u ∂u + 3 + 2 = 0 subject to: u(x, 0) = f (x) and (x, 0) = g(x) 2 2 ∂x ∂x∂t ∂t ∂t 1. Show that this is a hyperbolic equation, and obtain the canonical variables that would transform this equation to ∂2u =0 ∂ξ∂η whose general solution is given by u(x, t) = φ(ξ(x, t)) + ψ(η(x, t)). 2. Show that the same general solution can be obtained by treating the differential equation as a reducible type. 3. Plot the d’Alembert solution for the case where 4  ζ (αi , βi , γi , x) g(x) = sech(x) and f (x) = i=1

where

ζ (α, β, γ, x) =

 γ 1 + tanh (αx + β) 2

and

i 1 2 3 4

αi 1 1 1 1

βi 4 −4 4 10

γi 1 −1 −0.5 0.5

for −15 ≤ x ≤ 15 and 0 ≤ t ≤ 2.5. E11.4. Solve the wave equation for x ≥ 0, ∂2u ∂2u = ∂t2 ∂x2 subject to ⎫ u (x, 0) = e−2x ⎬ ∂u 1 for x ≥ 0 ; (x, 0) = ⎭ 4x ∂t 1+e

u(0, t) =

9 − e−4t 8

t≥0

11.6 Exercises

445

(Hint: See Section K.1.2 for the solution of the wave equation that includes a Dirichlet boundary condition.) E11.5. Obtain the displacement u(x, t) for a string fixed at two ends described by the following wave equation 2 ∂2u 2∂ u = γ ∂t2 ∂x2 subject to u(x, 0) = f (x), ∂u/∂t(x, 0) = g(x) and u(0, t) = u(L, t) = 0. Plot the solution for the case where L = 1, g(x) = 0 and + 0.2x 0 ≤ x ≤ 0.5 f (x) = 0.2(1 − x) 0.5 < x ≤ 1

E11.6. Consider the diffusion equation with nonhomogeneous boundary conditions: ∂u ∂2u = α2 2 ∂t ∂x subject to u(0,  t) = 1  ∂u u(x, 0) = x + cos (πx) and  2u(1, t) + = 1 ∂x x=1 1. Show that the approach in Section 11.4 can transform the problem to a homogeneous partial differential equation with homogenous boundary conditions given by ∂U ∂2U = α2 2 ∂t ∂x with conditions 4 U(x, 0) = cos (πx) − 1 + x 3

and

U(0,  t) ∂U  2U(1, t) + ∂x x=1

=

0

=

0

where u(x, t) = U(x, t) + 1 − x/3. 2. Solve the problem and plot the solutions. E11.7. Use the Sturm-Liouville approach given in Section 11.3.3 to show that the required weighing functions for the orthogonality of the solutions of the following eigenfunction equations are given by: 1. Legendre polynomials:   du d + n (n + 1) u = 0 → r(x) = 1 (1 − x)2 dx dx 2. Associated Legendre polynomials:     d m2 2 du + n (n + 1) − u = 0 → r(x) = 1 (1 − x) dx dx 1 − x2 3. Chebyshev polynomials:   d2 u du 1 1 − x2 + n 2 u = 0 → r(x) = √ −x 2 dx dx 1 − x2 4. Laguerre polynomials: x

du d2 u + nu = 0 → r(x) = e−x + (1 − x) 2 dx dx

446

Linear Partial Differential Equations

5. Associated Laguerre polynomials: d2 u du + (k + 1 − x) + nu = 0 → r(x) = e−x xk 2 dx dx 6. Hermite polynomials: x

d2 u du 2 − 2x + 2n (n + 1) u = 0 → r(x) = e−x 2 dx dx 7. Spherical Bessel functions:   d2 ( + 1) u = 0 → r(x) = 1 (xu) + x − dx2 x E11.8. Let f (x) be a continuous function of x. Show that for a set of functions   φn (x) = cos n f (x) a Sturm-Liouville differential eigenfunction equation (in expanded form) for these functions can be constructed as follows  −2 2  2   −3 df d φn d f df dφn − + n 2 φn = 0 2 2 dx dx dx dx dx Using this result, find a series approximation based on φn (x) using f (x) = x2 to approximate ⎧ ⎨ (x) for 0.25 ≤ x ≤ 0.5 h(x) = ⎩ (1 − x) for 0.5 < x ≤ 0.75 E11.9. A well-known equation for financial option pricing is the Black-Scholes equation for a ≤ x ≤ b and 0 ≤ t ≤ T given by   ∂u 1 ∂2u ∂u = σ2 x2 2 + r x −u ∂t 2 ∂x ∂x subject to u(a, t) = u(b, t) = 0

u(x, 0) = u f (x)

and

where u(x, t) is the value of the option, x is the value of the underlying asset, t is the time from expiry of option, r is the risk-free interest rate, σ is the volatility of the asset, and u f (x) is the final payoff. The boundaries a and b are barriers under which the option becomes worthless. Obtain the solution and plots for r = 1, σ = 2, a = 0.5, b = 2 and ⎧ ⎪ if 0.5 ≤ x ≤ 1 ⎨ 10x − 5 u f (x) = −5x + 10 if 1 ≤ x ≤ 2 ⎪ ⎩ 0 otherwise (Hint: Using separation of variables, an eigenvalue problem that is a special case of Example 11.9 will result.) E11.10. Obtain the solution to the following convective-diffusion equation: D

∂ 2 u ∂u ∂u + = ∂x2 ∂x ∂t

subject to: u(0, t) u(1, t)

= =

sin (ωt) 0

and

u(x, 0) = 0

Plot the solution for the case where D = 0.1 and ω = 2π, 0 ≤ t ≤ 1.

11.6 Exercises

447

E11.11. For the reaction-diffusion equation with first-order reaction given by ∂u ∂2u = D 2 − ku ∂t ∂x show that this can be converted to a simple diffusion problem for q(x, t) by using a simple transformation u(x, t) = e−kt q(x, t). E11.12. Consider the Fokker-Planck equation with constant coefficients given by ∂2u ∂u ∂u = +μ 2 ∂x ∂t ∂x where D and μ are constant diffusion and drift coefficients, respectively. Let u = eAx eBt q(x, t). Find the values of A and B that would transform the original equation into a simple diffusion equation for q(x, t), that is, for some α, D

∂q ∂2q = α2 2 ∂t ∂x E11.13. The Nusselt problem models the temperature for a laminar flow of a fluid through a pipe. After normalization, we have  2  ∂u ∂ u 1 ∂u = + 0 ≤ z ≤ L and 0 ≤ r ≤ 1 ∂z ∂r2 r ∂r subject to ∂u u(1, z) = uW (0, z) = 0 and u(r, 0) = u0 ∂r Solve this equation using the method of separation of variables. Plot the solution for the case with uW = 5 and u0 = 10. E11.14. Consider the potential equation for the temperature u of a sphere of radius. Assume cylindrical symmetry for the surface temperature, u(1, θ, φ) = f (θ); then we have only dependence with r and θ. Thus ∂2u ∂u ∂ 2 u cos θ ∂u + 2r + 2 + =0 ∂r2 ∂r ∂θ sin θ ∂θ where 0 ≤ θ ≤ π and 0 ≤ r ≤ 1. ∇ 2u = 0

→

r2

1. Let u = R(r)(θ); then show that separation of variables will yield the following ordinary differential equations: r2

d2 R dR + 2r − μR dr2 dr

=

0

(11.88)

d2  cos θ d + + μ = 0 (11.89) dθ2 sin θ dθ 2. Letting x = cos θ and μ = n(n + 1), show that (11.89) reduces to the Legendre equation (cf. Section 9.2), that is, d2  d − 2x + n(n + 1) = 0 2 dx dx whose solutions (considering Qn (0) = ±∞) are (1 − x2 )

n (θ) = Pn (cos θ) where Pn is the Legendre polynomial of order n defined in (I.31). 3. Using μ = n(n + 1), solve for R(r) under the condition that R(0) is finite. 4. Obtain the solution for u(r, θ) under the additional boundary conditions u(1, θ) = 1 − cos2 (θ), 0 ≤ θ ≤ π and plot the solution.

448

Linear Partial Differential Equations

E11.15. Consider the heat conduction in a solid sphere of radius R = 1 initially at temperature u(r, 0) = f (r) that is suddenly immersed in a constant bath temperature of us . Assume that the heat transfer is symmetric around the origin, that is, u = u(r, t) then the thermal diffusion equation is given by  2  ∂u ∂ u 2 ∂u 2 = α∇ u = α + (11.90) ∂t ∂r2 r ∂r subject to    ∂u  ∂u   u(r, 0) = f (r) = h (us − u) =0 r=R ∂r  ∂r  r=R

r=0

1. Show that with u = q/r, (11.90) can be recast in terms of q to yield ∂q ∂2q =α 2 ∂t ∂r 2. Solve for u(r, t) for the case where f (r) = u0 = 0, uS = 1, α = 1 and h = 5 and obtain u(r = 0, t). E11.16. Use separation of variables to derive the solution of ∂2U ∂2U + =0 ∂x2 ∂y2 x(x − 1) subject to U(0, y) = U(1, y) = 0, U(x, 0) = U(x, 1) = to show that 2    ∞ 4  sin (nπx) 1 − cosh(nπ) U(x, y) = − 3 cosh(nπy)+ sinh(nπy) π n3 sinh(nπ) n=1,3,5,...

E11.17. Solve the following nonhomogeneous partial differential equation using the eigenfunction expansion approach: ∂u ∂2u = 2 − 3ex−2t ∂t ∂x

0 ≤ x ≤ 1 and

t≥0

subject to u(0, t) = e−2t

,

u(1, t) = e1−2t

and

u(x, 0) = ex

Plot the solution and compare with the exact solution u(x, t) = ex−2t . Hint: First, use the approach given in Section 11.4 to obtain a splitting of the solution as u(x, t) = U(x, t) + S(x, t), such that   ∂2U ∂U ∂S ∂ 2 S x−2t = − 3e − − 2 ∂t ∂x2 ∂t ∂x subject to U(0, t) = 0 ,

U(1, t) = 0

and

U(x, 0) = ex − S(x, 0)

Then show that the eigenfunction approach solution should yield U(x, t) = ∞ n=1 T n (t) sin(nπt) where T n (t) is the solution of the ordinary differential equation: dT n + (nπ)2 T n = ξn e−2t subject to T n (0) = p n dt where   1 2 n pn = (−1) e − 1 nπ [(nπ)2 + 1]   ξn = (p n ) (nπ)2 − 2

11.6 Exercises

449

E11.18. Another approach to the solution of the Poisson equation ∂2u ∂2u + 2 = f (x, y) ∂x2 ∂y with homogeneous boundary conditions: u(x, 0) = u(x, 1) = u(0, y) = u(1, y) = 0, is to use eigenfunctions of the form φmn (x, y) = sin(nπx) sin(mπy) Thus we can approximate both u(x, y) and f (x, y) by ∞  ∞  ∞ ∞   u(x, y) = αmn φmn (x, y) and f (x, y) = βmn φmn (x, y) m=1 n=1

m=1 n=1

1. Using these approximations, obtain the equations needed to evaluate the coefficients αmn and βmn . 2. Apply these approximations to the case where f (x, y) = −1 and compare them to the results given in Example 11.12. E11.19. Consider the case of diffusion with linearly varying diffusivity given by ∂2u ∂u = 2 ∂x ∂t Use the similarity transformation approach to solve the equation and obtain a plot for x ≥ 0 and t ≥ 0 based on the following conditions: u(x, 0) = 1, x ≥ 0 and u(1, t) = 0. x

E11.20. The chemical vapor deposition reaction (CVD) that is fed by a diffusionlimited laminar flow between two parallel plates can be modeled approximately by7 ∂u Q ∂2u = ∂x y ∂y2 subject to: u(x, 0) = 0, u(0, y) = u0 , lim u(x, y) = u0 and lim u(x, y) = 0, y→∞

x→∞

where u is the metal concentration, y is the distance from wall, x is the DB distance from the entrance along the laminar flow, and Q = , with D, 2vmax vmax , and B as the diffusion coefficient, maximum velocity, and midpoint distance between the two parallel plates, respectively. The solution will be valid only near a plate because the approximation has been obtained via linearization around y = 0. Obtain and plot the solution using a similarity transformation approach, with Q = 10.

7

Based on an example in R. G. Rice and D. D. Do, Applied Mathematics and Modeling for Chemical Engineers, John Wiley & Sons, New York, 1995, pp. 415–420.

12

Integral Transform Methods

In this chapter, we discuss the integral transform methods for solving linear partial differential equations. Although there are several types of transforms available, the methods that are most often used are the Laplace transform methods and the Fourier transform methods. Basically, an integral transform is used to map the differential equation domain to another domain in which one of the dimensions is reduced from derivative operations to algebraic operations. This means that if we begin with an ordinary differential equation, the integral transform will map the equation to an algebraic equation (cf. Section 6.7). For a 2D problem, the integral transforms should map the partial differential equation to an ordinary differential equation, and so on. We begin in Section 12.1 with a very brief introduction to general integral transforms. Then, in Section 12.2, we discuss the details of Fourier transforms, its definition, some particular examples, and then the properties. Surprisingly, the initial developments of Fourier transforms are unable to be applied to some of the most useful functions, including step function and sinusoidal functions. Although there were several ad hoc approaches to overcome these problems, it was not until the introduction of the theory of distributions that the various ad hoc approaches were unified and gained a solid mathematical grounding. This theory allows the extension of classical Fourier transform to handle the problematic functions. We have located the discussion of theory of distributions in the appendix, and we include the details of the Fourier transform of the various “difficult” functions in the same appendix. Some of properties of Fourier transforms are then explored and collected in Section 12.2.2. We have kept these properties to a minimum, with a focus on solving differential equations. Then, in Section 12.3, we apply Fourier transform to solve partial differential equations. As some authors have noted, Fourier transform methods are most useful in handling infinite domains. Next, starting in Section 12.4, we switch our attention to the Laplace transforms. We view Laplace transforms as a special case of Fourier transforms. However, with Laplace transform, one can apply the technique on dimensions that are semi-infinite, such as time variables. Thus there are strong similarities between the Laplace and Fourier transforms of functions as well as in their properties. There are also significant differences. For one, the handling of several functions such as step, sinusoidal, exponential, Bessel, and Dirac delta functions are simpler to obtain. The inversion of Laplace transforms, however, can be quite complicated. In several cases, using a 450

12.1 General Integral Transforms

451

table of Laplace transforms, together with separation to partial fractions, are sufficient to obtain the inverse maps. In general, however, one may need to resort to the theory of residues. We have included a review of the residue methods in the appendix. Then, in Section 12.5, we apply Laplace transforms to solve some partial differential equations. Finally, in Section 12.6, we include a brief section to show how another technique known as method of images can be used to extend the use of either Fourier or Laplace methods to semi-infinite or bounded domains.

12.1 General Integral Transforms Along with Fourier and Laplace transforms, other integral transforms include Fourier-Cosine, Fourier-Sine, Mellin, and Hankel transforms. In this section, we discuss some general results about integral transforms before we focus on Fourier and Laplace transforms. For a review of complex integration, we have included a brief discussion in the appendix as Section L.2 to cover the theorems and methods needed for computing the transforms. Definition 12.1. For a given function f (x), the integral  IK,a,b [ f (x)] =

b

K (p, x) f (x) dx

(12.1)

a

is called the integral transform of f , where K(p, x) is called the kernel of the transform and p is called the transform variable of the integral transform. The limits of integration a and b can be finite or infinite. A list of different useful integral transforms is given in Table 12.1. For our purposes, we take integral transforms simply as a mapping of the original function based on variable x to another function based on a new variable p . The expectation is that in the new domain, convenient properties are obtained, by which the analysis becomes more manageable. For instance, with Laplace transforms, the original problem could be a linear time-invariant linear differential equations. After taking Laplace transforms of these equations, the differential equations are replaced by algebraic equations. Based on the properties of integrals, integral transforms immediately satisfy the linearity property, that is, with constants α and β,  IK,a,b [αf (x) + βg(x)]

b

=

K (p, x) (αf (x) + βg(x)) dx

a



=

b

α a

=



b

K (p, x) f (x) dx + β

K (p, x) g(x) dx a

αIK,a,b [ f (x)] + βIK,a,b [g(x)]

Once the analytical manipulation in the transform domain is finished, an inverse transformation is needed to obtain a solution in the original domain. Thus another important criteria for the choice of integral transforms is how easy the inverse transformations can be evaluated. In most integral transforms, such as those given

452

Integral Transform Methods Table 12.1. Various integral transforms Name

Transform

Mellin

M [f (x)] =



∞

f (x)x p −1 dx

0

7 Fourier Cosine

7 Fourier Sine

Fs [ f (x)] = 

Fourier

F [f (x)] =

Laplace

L [f (x)] =



2 π

Fc [ f (x)] =

2 π

f (x) cos(xp ) dx 0



∞

f (x) sin(xp ) dx 0

∞

f (x)e−ixp dx

−∞



∞

∞

f (x)e−xp dx

0

 Hankel transform

∞

Hν [ f (x)] =

f (x)xJ ν (xp ) dx 0

in Table 12.1, the inverse transformation is chosen to be another integral transform, that is,  f (x) =

 b

 a

 p ) dp (IK,a,b [ f ]) K(x,

(12.2)

 p ) is the kernel of the inverse integral transform. In general, a =  where K(x, a,   p ). In fact, in some cases, a and b could be real numbers, b = b, and K(x, p ) = K(x,  p ), then we call whereas  a and  b are complex numbers. In case K(x, p ) = K(x, K(x, p ) a Fourier kernel. For example, the kernels of the Fourier-cosine transforms and Fourier-sine transforms are Fourier kernels.1

12.2 Fourier Transforms In this section, we first derive the Fourier transform and then obtain some important properties. Later, we apply the operation of Fourier transform to the solution of partial differential equations.

12.2.1 Definition and Notations The Fourier series of a periodic function f (x) with period T is f (x) = a0 +

∞ '  =1

1

 a cos

  ( 2π 2π x + b sin x T T

(12.3)

Ironically, the kernel of the Fourier transform, K(x, p ) = e−ixp , is not a Fourier kernel; that is, the  p ) = eixp /2π. kernel of the inverse Fourier transformation is K(x,

12.2 Fourier Transforms

453

where a and b are the Fourier coefficients given by 



T/2

f (t) cos (2πt/T ) dt

a =

T/2

f (t) sin (2πt/T ) dt

−T/2  T/2

b =

and

−T/2  T/2

2

−T/2

cos (2πt/T ) dt

−T/2

sin2 (2πt/T ) dt

With T = 2πτ, the coefficients become a0

=

a

=

b

=

 πτ 1 f (t)dt 2πτ −πτ    πτ 1 t f (t) cos dt πτ −πτ τ    πτ 1 t f (t) sin dt πτ −πτ τ

(12.4)

An efficient way to solve for the coefficients is through the use of fast Fourier transforms (FFT). The FFT method is described in Section L.1 as an appendix. Substituting (12.4) into (12.3), f (x) =



1 2πτ

πτ −πτ

f (t)dt +

   πτ ∞  1  f (t) cos (x − t) dt πτ −πτ τ =1

Letting (1/τ) = ω and /τ = ω,

ω f (x) = 2π



πτ −πτ

∞

1 f (t)dt + π =1

'

π/ ω −π/ ω

( f (t) cos (ω(x − t)) dt ω

(12.5)

By taking the limit of f (x) as ω approacheszero, the summation in (12.5) becomes ∞ an integral. Also, with the assumption that ( −∞ f (t)dt < ∞), the first term in (12.5) will become zero. We end up with the formula known as the Fourier integral equation,   1 ∞ ∞ f (t) cos (ω(x − t)) dt dω (12.6) f (x) = π 0 −∞ Assuming for now that (12.6) is valid for the given function f (x), we could proceed further by obtaining an alternative form of (12.6) based on Euler’s identity. Taking the integral of Euler’s identity,  m  m  m i(x−t)ω e dω = cos ((x − t)ω) dω + i sin ((x − t)ω) dω −m

−m



=

m

2

−m

cos ((x − t)ω) dω

0

or  0

m

cos ((x − t)ω) dω =

1 2



m −m

ei(x−t)ω dω

(12.7)

454

Integral Transform Methods

Substituting (12.7) into (12.6) while switching the order of integration, ' ∞ (  1 ∞ f (x) = f (t) cos (ω(x − t)) dω dt π −∞ 0 ' ∞ (  ∞ 1 i(x−t)ω = f (t) e dω dt 2π −∞ −∞ ' ∞ (  ∞ 1 ixω −itω = e f (t)e dt dω 2π −∞ −∞

(12.8)

Equation (12.8) can now be deconstructed to yield the Fourier transform pair. Definition 12.2. For a given function f (x), the operator F acting on f (x) given by  ∞ f (t)e−itω dt (12.9) F (ω) = F [ f ] = −∞

is called the Fourier transform of f (t). For a given function F (ω), the operator F −1 acting on F (ω) given by  ∞ 1 f (x) = F −1 [F ] = F (ω)eixω dω (12.10) 2π −∞ is called the inverse Fourier transform of F (ω). Thus the kernel of the Fourier transform is K(x, ω) = e−ixω , whereas the inverse  ω) = (1/2π)eixω ; that is, the signs in the exponential power in kernel is given by K(x, both kernels are opposites of each other.2 The Fourier integral equation, (12.6), is not valid for all types of functions. However, (12.6) is valid for functions that satisfy the Dirichlet conditions and are integrable, that is,  ∞    f (t) dt ≤ ∞ (12.11) −∞

which in fact generalizes (12.6) to the following formulation:     1   + 1 ∞ ∞ f x + f x− = f (t) cos (ω(x − t)) dt dω 2 π 0 −∞

(12.12)

Details of Dirichlet conditions and the corresponding Fourier integral theorem, Theorem L.8, can be found in an appendix of this chapter, under Section L.3. Included in the discussion is the proof that when Dirichlet and integrability conditions are satisfied, the interchangeability of integration sequence used to derive (12.8) is indeed allowed. The evaluation of the Fourier and inverse Fourier transforms usually requires the rules and methods of integration of complex functions. A brief (but relatively extensive) review of complex functions and integration methods are included in an appendix under Section L.2. Included in Section L.2 are the theory of residues to 2

There are several versions for the definition of Fourier transform. One version switches the sign of the exponential. Other versions have different coefficients. Because of the existence of different definitions, it is crucial to always determine which definition was chosen before using any table of Fourier transforms.

12.2 Fourier Transforms 1.5

455

2.5

F(ω)=2 sin(ω)/ω f(t)=H(1−|t|)

2

1

F(ω)

f(t)

1.5

0.5

1

0.5

0 0

−0.5 −2

−1.5

−1

−0.5

0

0.5

1

1.5

2

−0.5 −30

−20

−10

t

0

ω

10

Figure 12.1. A plot of the square pulse function and its corresponding Fourier transform.

evaluate contour integrals and extensions needed to handle integrals with complex paths of infinite limits, regions with infinite number of poles, and functions with branch cuts. Examples addressing Fourier transforms are also included. EXAMPLE 12.1.

Consider the square pulse given by   + 0 if |t| > a f (t) = H a − |t| = 1 if |t| ≤ a

(12.13)

where a > 0 is a constant. Then the fourier transform is given by  ∞      H a − |t| e−iωt dt F H a − |t| =  =

−∞ a −a

e−iωt dt = 2

sin (aω) ω

Plots of (12.13) and (12.14) are shown in Figure 12.1. The inverse Fourier transform of the square pulse is given by  ∞      1 = F −1 H b − |ω| H b − |ω| eiωt dω 2π −∞  b 1 sin (bt) = eiωt dω = 2π −b πt

(12.14)

(12.15)

Consider the function, F (ω) = e−|ω| . Using the definition given in (12.10), the inverse Fourier transform is given by ' (  ∞ 1 = F −1 e−|ω| e−|ω| eiωx dω 2π −∞ 9 8  ∞ 0 1 (ix+1)ω (ix−1)ω = e dω + e dω 2π −∞ 0   1 1 1 1 1 = − = (12.16) 2 2π ix + 1 ix − 1 π x +1

EXAMPLE 12.2.

20

30

456

Integral Transform Methods

Unfortunately, some useful and important functions, such as the unit step function, sine, cosine, and some exponential functions, do not satisfy the required conditions of integrability. The definition of Fourier transforms needed to be expanded to accommodate these important functions. Before the 1950s, the problematic functions were treated on a case-by-case basis using arguments that involve taking limits of some parameters to approach either 0 or infinity. Many of the results using these approaches were verified by successful physical applications and practice, especially in the fields of physics and engineering. As a result of their successes, these approaches remain valid and acceptable to most practicioners, even at present time. Nonetheless, these approaches lacked crucial mathematical rigor and still lacked generality in their approach. The biggest contention was centered on the fact that the “Dirac delta function” did not satisfy the conditions required of functions. With the introduction of distribution theory by Laurent Schwartz, most of the mathematical rigor needed was introduced. The theory allowed the definition of the Dirac delta function as a new object called a distribution. A subset of distributions called tempered distributions was subsequently constructed. Using tempered distributions, a generalized Fourier transform was formulated, and this allowed a general approach to handle Fourier transforms of functions such as sines, cosines, and so forth. Fortunately, the previous approaches using limiting arguments were proven to be equivalent to the methods of distribution theory. Thus the previous methods were validated with more solid mathematical basis. More importantly, a general approach had become available for problems in which the limiting argument may be difficult to perform or justify. A short introduction to distribution theory and delta distributions is included in Section L.4 as an appendix. Included in Section L.4 are general properties and operations of distributions and specific properties and operations of delta distributions. A discussion of tempered distribution and its application to the formulation of generalized Fourier transform continues in Section L.5 as another appendix. Included in Section L.5 are the definition of the generalized Fourier transforms; the evaluation of Fourier transforms of sines, cosines, unit step functions, and delta distributions using the methods of generalized Fourier transforms; and additional properties such as the Fourier transforms of integrals. With the formulation of generalized Fourier transforms, we specifically refer to the original definitions given in (12.9) and (12.10) as the classic Fourier transform and the classic inverse Fourier transform, respectively. As shown in Section L.5, the computations used for Fourier transforms of tempered distributions still need both these definitions. Thus we use the term “Fourier transform” to imply generalized Fourier transform, because the classic Fourier transform is already included in the generalized forms. This means that the integral formulas of the classic Fourier and classic inverse Fourier transforms are used for most evaluations, until a problem with integrability or the presence of delta distributions occurs, at which point the methods of generalized Fourier transforms using tempered distributions are applied.

12.2.2 Properties of Fourier Transforms We now list some of the important properties of Fourier transforms and inverse Fourier transforms. In the following, we use F (ω) to mean F [f (t)]:

12.2 Fourier Transforms

457

1. Linearity. Let a and b be constants, F [af (t) + bg(t)] F

−1

=

[aF (ω) + bG(ω)]

aF [ f (t)] + bF [g(t)]

=

aF

−1

[F (ω)] + bF

−1

(12.17) [G(ω)]

(12.18)

2. Shifting.  F [f (t − a)]

=

∞

 = =

F

−1

[F (ω − b)]

f (t − a)e−iωt dt

−∞ ∞

f (τ)e−iω(τ+a) dτ

−∞

e−iaω

=

1 2π

=

eibt

∞

f (τ)e−iωτ dτ = e−iaω F [f (t)]

−∞



1 2π

=



∞

−∞



∞

−∞

1 2π



(τ = t − a) (12.19)

F (ω − b)eiωt dω F (ν)ei(ν+b)t dν ∞

−∞

(ν = ω − b)

F (ν)eiνt dν = eibt F −1 [F (ω)] (12.20)

3. Scaling. Let a > 0,  F [f (at)]

=

−∞

=

1 a

=

1 a

For a < 0,

 F [ f (at)]

∞

=







=

1 − a

∞

−∞

−∞

1 a

∞

−∞

∞

=

f (at)e−iωt dt f (τ)e−iω(τ/a) dτ f (τ)e−i(ω/a)τ dτ =

(τ = at) 1 ω F a a

f (at)e−iωt dt −infty

∞



∞

−∞

f (τ)e−iω(τ/a) dτ

(τ = at)

1 ω f (τ)e−i(ω/a)τ dτ = − F a a

Combining both results, F [ f (at)] =

1 ω F |a| a

(12.21)

In particular, with a = −1, we have F [ f (−t)] = F (−ω)

(12.22)

458

Integral Transform Methods

4. Derivatives. ' F

df dt

(



∞

=

df −iωt e dt −∞ dt (∞  f (t)e−iωt + iω

=

iωF [ f (t)]

=

∞

−∞

−∞

f (t)e−iωt dt

(after integration by parts)

(after setting values at limits to zero)

(12.23)

or in general, we can show by induction that ' F

dn f dtn

( = (iω)n F [ f (t)]

(12.24)

For the inverse Fourier transform, we have F −1

'

dF dω

(

1 2π



∞

dF iωt e dω −∞ dω  (∞   ∞ 1 F (ω)eiωt − it F (ω)eiωt dω 2π −∞ −∞

= = =

−itF −1 [F (ω)]

=

(−it) f (t)

(12.25)

or in general, F −1

'

dn F dωn

( = (−it)n f (t)

(12.26)

5. Integrals. ' F

t

−∞

( 1 f (η)dη = F (ω) + πδ(ω)F (0) iω

(12.27)

The derivation of (12.27) is given in the appendix under Section L.5.2 and requires the use of generalized Fourier transforms. 6. Multiplication and Convolution. For any two functions f (τ) and g(τ), the convolution operation, denoted by ‘∗’, is defined as  Convolution(f, g) = [ f ∗ g] (t) =

∞

−∞

f (t − τ)g(τ)dτdτ

The Fourier transform of [ f ∗ g] (t) is    F [ f ∗ g](t) =

∞

−∞

e−iωt



∞ −∞

f (t − τ)g(τ)dτdt

(12.28)

12.3 Solution of PDEs Using Fourier Transforms

Let η = t − τ, then   F [ f ∗ g](t)

 =

∞

−∞

 =

∞

−∞

e−iω(η+τ)



459

∞

f (η)g(τ)dτdη

−∞

e−iωη f (η)dη



=

    F f (η) F g(τ)

=

F (ω)G(ω)

∞

−∞

e−iωτ g(τ)dτ

(12.29)

Note that the convolution operation is commutative, that is, [f ∗ g](t) = [g ∗ f ](t). Going the other way, we can obtain the inverse Fourier transforms of convolutions.   F −1 [F ∗ G] (ω)

=

= =

1 2π



∞ −∞

 eiωt

∞ −∞

F (ω − ν)g(ν)dνdω

let σ = ω − ν  ∞  ∞ 1 it(σ+ν) e F (σ)g(ν)dνdσ 2π −∞ −∞     ∞  ∞ 1 1 −itσ itν 2π e F (σ)dσ e G(ν)dν 2π −∞ 2π −∞

=

2πF −1 [F (σ)] F −1 [G(ν)]

=

2πf (t)g(t)

(12.30)

When it comes to applications of Fourier transform to solve differential equations, the dual versions of (12.29) and (12.30) are more frequently used, that is,   = [ f ∗ g](t) (12.31) F −1 F (ω)G(ω)   F f (t)g(t)

=

1 [F ∗ G](ω) 2π

(12.32)

These six properties are summarized in Table 12.2. Using either direct computation or implementation of the properties of Fourier transforms, a list of the Fourier transforms of some basic functions is given in Table 12.3. In some cases, the Fourier transforms can also be obtained by using known solutions of related differential equations. One such case is the Fourier transform of the Airy function, Ai(x) (see Exercise E12.1).

12.3 Solution of PDEs Using Fourier Transforms In applying Fourier transforms for the solution of partial differential equation, one needs to carefully choose the independent variable used for transformation. For

460

Integral Transform Methods Table 12.2. Properties of fourier and inverse fourier transforms

1. Linearity

2. Shifting

3. Scaling

F [af (t) + bg(t)]

=

aF [ f (t)] + bF [g(t)]

F −1 [aF (ω) + bG(ω)]

=

aF −1 [F (ω)] + bF −1 [G(ω)]

F [ f (t − a)]

=

e−iaω F [f (t)]

F −1 [F (ω − b)]

=

eibt F −1 [F (ω)]

F [ f (at)] =

1 ω F |a| a

'

( dn f = (iω)n F [f (t)] dtn ' n ( d F F −1 = (−it)n F −1 [F (ω)] dωn ' t ( 1 F f (η)dη = F (ω) + πδ(ω)F (0) iω −∞   = [f ∗ g](t) F −1 F (ω)G(ω)   1 F f (t)g(t) = [F ∗ G](ω) 2π

F

4. Derivatives

5. Integrals

6. Convolution

instance, suppose the dependent variable is u = u(x, t), with −∞ < x < ∞ and 0 ≤ t ≤ ∞. Then the transform would make sense only with respect to x. When taking the Fourier transform with respect to x, the other independent variables are fixed during the transformations. For instance, with u(x, t), we define a new variable, U(ω, t) = F [u(x, t)] Some basic rules apply: 1. When taking derivatives with respect to the other independent variables, one can interchange the order of differentiation with the Fourier transform operation, for example, ( dk dk ∂k u(x, t) = F t)] = F U(ω, t) [u(x, ∂tk dtk dtk

'

(12.33)

Note that the partial derivative operation has been changed to an ordinary derivative. 2. When taking derivatives with respect to the chosen variable under which Fourier transforms will apply, say x, the derivative property of Fourier transforms can be used, that is, ' F

( ∂k u(x, t) = (iω)k F [u(x, t)] = (iω)k U(ω, t) ∂xk

(12.34)

12.3 Solution of PDEs Using Fourier Transforms Table 12.3. Fourier transforms of some basic functions f (t) +

if t ≤ 0 if t > 0

0 1

F [f (t)]

Some remarks

πδ (ω) + [1/ (iω)]

See Example L.11

2/ (iω)

See Example L.11

2 sin(aω)/ω

See Example 12.1

1

H (t) =

2

sgn(t) =

3

H (a − |t|) =

4

sin(bt)/t

  πH b − |ω|

See Example 12.1

5

δ(t − a)

eiaω

See Example L.9

6

  a/ t2 + a2

πe−a|ω|

See Example 12.2

7

1

2πδ(ω)

See Example L.10

8

eiat

2πδ(ω − a)

See Example L.10

e−|a|t

  √ 2 π/| a | e−ω /(4|a|)

See Example L.4

+

−1 1 +

9

2

if t < 0 if t > 0 0 1

if |t| > a if |t| ≤ a

10

cos(at)

  π δ (ω + a) + δ (ω − a)

See Example L.10

11

sin(at)

  iπ δ (ω + a) − δ (ω − a)

See Example L.10

However, this relationship is true only if t=∞  iωt  =0 u(x, t)e  t=−∞

One sufficient case is when u(x, t) → 0 as |x| → ∞. The general approach to solving the partial differential equation for u(x, t) can be outlined as follows: 1. Take the Fourier transform of the partial differential equation with respect to x and reduce it to an ordinary differential equation in terms of U(ω, t), with t as the independent variable. 2. Take the Fourier transform of the conditions at fixed values of t, for example, ' ( ∂u d U(ω, 0) = F (x, 0) U(ω, 0) = F [u(x, 0)] and dt ∂t 3. Solve the ordinary differential equation for U(ω, t). 4. Take the inverse Fourier transform of U(ω, t). In several applications, one could use the table of Fourier transforms in conjunction with the properties of Fourier transforms, such as the shifting theorem and convolution theorem, to obtain a general solution.

461

462

Integral Transform Methods

Fourier transforms approach to obtain d’Alembert solution. For the one-dimensional wave equation given by

EXAMPLE 12.3.

∂2u 1 ∂2u − 2 2 =0 2 ∂x c ∂t subject to initial conditions

−∞≤x≤∞

∂u (x, 0) = g(x) ∂t We first take the Fourier transform of both sides, ' 2 ( ∂ u 1 ∂2u 1 d2 2 F − = U(ω, t) − U = (iω) ∂x2 c2 ∂t2 c2 dt2 u (x, 0) = f (x)

and

d2 U + (cω)2 U dt2

=

0 0

whose solution can be put in terms of eicωt and e−icωt as U(ω, t) = Aeicωt + Be−icωt Applying the initial conditions, we have F [ f (x)] = A + B or



A B



1 = 2

and 

1 1

F [g(x)] = Aicω − Bicω

1/(icω) −1/(icω)



F [f ] F [g]



Then,   1  icωt 1  icωt e + e−icωt F [f (x)] + e − e−icωt F [g(x)] 2 2icω Next, take the inverse Fourier transforms of the various terms while applying the shifting theorem (12.19), ' ( 1 icωt 1 F −1 = e F [ f (x)] f (x + ct) 2 2 ' ( 1 −icωt 1 F −1 = e F [ f (x)] f (x − ct) 2 2 8 9 8 ' x ( 9 1 icωt 1 −1 −1 icωt F e F [g(x)] = F g(η)dη e F 2icω 2c −∞     π F −1 eicωt δ(ω) − F [g] 2c ω=0   x+ct  1 1 = g(η)dη − F [g] 2c −∞ 4c ω=0 8 9   x−ct  1 −icωt 1 1 F −1 e F [g(x)] = g(η)dη − F [g] 2icω 2c −∞ 4c ω=0

U(ω, t) =

Adding all the terms together, we obtain d’Alembert’s solution,   1 x+ct 1 f (x + ct) + f (x − ct) + u(x, t) = g(η)dη 2 2c x−ct

12.3 Solution of PDEs Using Fourier Transforms

463

10

u(x,y)

8

2 1.5

y

6

1

4 0.5

2

5

0 10

0

y

5

x

0 −5

0 –5

0

x

Figure 12.2. A surface plot of u(x, y) given by (12.35) together with the corresponding contour plot. EXAMPLE 12.4.

Consider the Laplace equation for the 2D half-plane, y ≥ 0, ∂2u ∂2u + 2 =0 ∂x2 ∂y

subject to u(x, 0)

=

f (x) = H (1 − |x|)

u(x, y)

→

0 as |x| → ∞ and y → ∞

Taking the Fourier transform of both sides with respect to x, (−iω)2 U(ω, y) +

d2 U =0 dy2

whose solution is given by U(ω, y) = Ae|ω|y + Be−|ω|y Because U(ω, y) < ∞ as y → ∞, we need A = 0. Applying the other boundary condition (keeping it in terms of f (x) in the meantime), U(ω, y) = F [ f ] e−|ω|y Using the convolution theorem given by (12.31) and item 6 in Table 12.3, we have   1 y u(x, y) = f (x) ∗ π x2 + y2   1 ∞ y 1 1 y = H (1 − |η|) dη = dη π −∞ (x − η)2 + y2 π −1 (x − η)2 + y2 ⎛ ⎞     1⎝ x−1 ⎠ x+1 = − Atan Atan (12.35) π y y Figure 12.2 shows a surface plot of u(x, y) given by (12.35), together with some curves at constant u values, which is shown separately as a contour plot. The level curves can be seen to be circles whose centers are located along the line x = 0.

5

464

Integral Transform Methods

12.4 Laplace Transforms In this section, we derive the Laplace transform as a special case of the Fourier transform. Then we obtain some important properties. We also include a subsection for the use of partial fractions to obtain inverse Laplace transforms of rational polynomial functions. Finally, we apply the Laplace transforms to the solution of partial differential equations, including some examples that combine with the application of Fourier transforms.

12.4.1 Definition and Notations Consider the function f˘ (x) given by + ˘f (x) = 0 e−ax f (x)

if x ≤ 0 if x > 0

(12.36)

where a is a positive real number. Assuming f˘ is integrable, we can apply it to the Fourier integral equation, (12.8), ' ∞ (  ∞ 1 f˘ (x) = eixω f˘ (t)e−itω dt dω 2π −∞ −∞ ' ∞ (  ∞ 1 H (x) e−ax f (x) = eixω f (t)e−(a+iω)t dt dω 2π −∞ 0 ( ' ∞  ∞ 1 (a+iω)x −(a+iω)t H (x) f (x) = e f (t)e dt dω (12.37) 2π −∞ 0 where H (x) is the unit step function. From (12.37), and letting s = a + iw, a ≥ 0, we can extract the integral transform pair called the Laplace transform and inverse Laplace transform. Definition 12.3. Let s be a complex variable whose real part is non-negative. For a given function f (t), the operator L acting on f (t) given by  ∞  f (t)e−st dt (12.38) f (s) = L [ f ] = 0

is called the Laplace transform of f . f (s) given by For a given function  f (s), the operator L−1 acting on     γ+i∞ 1  f = H (t) f (t) = L−1  f (s)est ds 2πi γ−i∞

(12.39)

is called the inverse Laplace transform of  f (s). The value of γ is a non-negative real number chosen to be greater than the real part of any singularities of  f (s).3 The kernel of a Laplace transform is given by K(t, ω) = e−st , whereas the inverse  s) = (1/2πi)est . During the evaluation of Laplace transforms, kernel is given by K(t, one often uses integration by parts and requires that values at the upper limit of t → ∞ are bounded. Alternatively, the direct evaluation of the inverse Laplace 3

The integral formula given in (12.39) is also known as the Bromwich integral.

12.4 Laplace Transforms

465

transforms via (12.39) may be obtained via the method of residues. Details of the residue theorem can be found in Sections L.2.2 and L.2.3.4 Briefly, we have 1 2πi



γ+i∞

γ−i∞

 f (s)est ds =

N 

  f (s)est Resz 

(12.40)

=1

inside a left semicircle contour of a vertical line and an arc large enough  composed  st  to contain the singularities of f e . The residues Resz (g), with z being a kth order poles of g(z), are given by Reszo (g) =

 1 dk−1  lim k−1 [z − zo ]k g(z) (k − 1)! z→zo dz

(12.41)

For the special case when the function g(z) is a rational function with a numerator function num(z) and denominator function den(z), that is, g(z) =

num(z) den(z)

such that a simple pole at (z = z0 ) is a root den(z), then via L’Hospital’s rule, we have    num(z)  Reszo (g) = (12.42)  d den(z)  dz z=z0 Laplace transform of tν . Consider the function f (t) = tν where ν > 0 is a real valued constant. Then  ∞  ∞  ∞  y ν  y  1 L [tν ] = e−st tν dt = e−y d = ν+1 e−y yν dy s s s 0 0 0

EXAMPLE 12.5.

 (ν + 1) sν+1 where (x) is the gamma function of x (cf. (9.7)). =

(12.43)

 √  Laplace transform of erfc 1/(2 t) . The error function, erf (x), and the complementary error function, erfc (x) are defined as  x  ∞ 2 2 2 −ν2 erf (x) = √ e dν and erfc (x) = √ e−ν dν (12.44) π 0 π x

EXAMPLE 12.6.

Note that erf (∞) = 1 and thus erf (x) + erfc (x) = 1 4

(12.45)

In practice, because the evaluation of the inverse Laplace transforms can be quite complicated, a table of Laplace transforms is usually consulted first.

466

Integral Transform Methods

 √  The Laplace transform of erfc 1/2 x is given by 8   9  ∞  ∞ 1 2 −st −ν L erfc √ =√ e √ e dνdt 2 x π 0 1/(2 t) After integration by parts, 8   9 1 L erfc √ = 2 x =

 ∞ −st −1/(4t) 1 e e dt √ √ 2s π 0 t t  ∞ 2 2 2 e−s/(4q ) e−q dq √ s π 0

1 with q = √ 2 t

Let g(s) be equal to the integral term, then  ∞ 2 2 g(s) = e−s/(4q ) e−q dq 0

dg ds 2

d g ds2



=

∞

− 

=

0 ∞

0

1 −s/(4q2 ) −q2 e e dq 4q2 1

42 q4

e−s/(4q ) e−q dq 2

2

By evaluating d2 g/ds2 through integration by parts,      d2 g 1 ∞ −s/(4q2 ) −q2 1 1 1 1 dg 1 = e e + dq = − + g ds2 s 0 8q2 4 s 2 ds 4 or d2 g 1 dg 1 s2 2 + s − sg = 0 ds 2 ds 4 This equation is reducible to a Bessel or modified Bessel equation. Using √ the methods described in Theorem 9.3, we have, after using g(0) = π/2 and |g(∞)| < ∞, √ √  √  √  g(s) = s1/4 AI1/2 s + BI−1/2 s = ae− s + be s √ π −√s = e 2 Combining the results, we have 8   9 1 1 √ L erfc √ = e− s (12.46) s 2 x

EXAMPLE 12.7.

(s) be given by Let F (s) = J ν (ξs) F J ν (s)

where ξ = {0, 1}. To find the inverse Laplace transform, we can use (12.39), that is,  γ+i∞   1 J ν (ξs) −1  F (x) = L ds est 2πi γ−i∞ J ν (s)

12.4 Laplace Transforms

467

and use the theory of residues to evaluate the complex integral. The poles of est will be the roots of J ν (s), except for s = 0, because it a removable singularity F as it is also the root of the numerator J ν (ξs). Thus let zn be the n th root of J ν (z). Using (12.40) and (12.42), plus the formula for the derivative of Bessel function given in (9.66), we have    J ν (ξzn ) (x) = ezn t ν L−1 F J ν (zn ) − J ν+1 (zn ) n=1,...,∞;zn =0 zn

12.4.2 Properties of Laplace Transforms We now list some of the important properties of Laplace transforms and inverse Laplace transforms. In the following, we use  f (s) to mean L [ f (t)]: 1. Linearity.   L af (t) + bg(t)   L−1 a f (s) + b g (s)

    a L f (t) + b L g(t)     f (s) + b L−1  a L−1  g (s)

= =

(12.47) (12.48)

where a and b are constants. Both (12.47) and (12.48) are immediate consequences of the properties of integrals. 2. Shifting.   L H (t − a) f (t − a)



∞

=

H (t − a) f (t − a)e−st dt

0



∞

= 0

−as

=

f (τ)e−s(τ+a) dτ 

∞

e

(τ = t − a)

f (τ)e−sτ dτ

0

  e−as L f (t)

=

  f (s − b) L−1 

=

1 2πi



γ+i∞

γ−i∞



γ+i∞

 f (s − b)est dω

=

1 2πi

=

ebt

=

  f (s) ebt L−1 

γ−i∞

1 2πi



(12.49)

 f (ν)e(ν+b)t dν

γ+i∞ γ−i∞

(ν = s − b)

 f (ν)eνt dν (12.50)

468

Integral Transform Methods

3. Scaling. Let Real(a) ≥ 0, a = 0 

  L f (at)

∞

= 0

= = =

1 a 1 a

 

f (at)e−st dtdt ∞

f (τ)e−s(τ/a) dτ

∞

f (τ)e−(s/a)τ dτ

0

1 f a

(τ = at)

0



s a

 (12.51)

4. Derivatives. 8 L

df dt

9



∞

df −st e dt dt 0 (∞  −st f (t)e +s

= =

  sL f (t)

f (t)e−st dt

(after integration by parts)

0

0

=

∞

− f (0)

(12.52)

or in general, we can show by induction that 8 L

dn f dtn

9



= sn L f (t)



−

n−1  k=0

⎛

⎞  k  d f sn−k−1 ⎝ k  ⎠ dt t=0

(12.53)

5. Integrals. 8  L

9

t



∞

=

f (τ)dτ 0



−st

t

e

f (τ)dτ

0

0

After integration by parts, 8 

9

t

L

f (τ)dτ 0

1 = s



∞

−st

f (t)e 0

1 dt = L s

'

( f (t)

(12.54)

6. Convolution. For Laplace transforms, we define a different convolution operator: 

t

Convolution(f, g) = [f ∗ g] (t) = 0

f (t − τ)g(τ)dτdτ

(12.55)

12.4 Laplace Transforms

469

Note that the limits for the convolution used in Laplace transform methods are from 0 to t, whereas the convolution for Fourier transform methods are from −∞ to ∞.5 The Laplace transform of a convolution is then given by ' t ( ' (  ∞ e−st f (t − τ)g(τ)dτ dt L f ∗g = 0

 =

∞

'

∞

'

∞

'

0

 = =

( e−st f (t − τ)g(τ)dτ dt

0

0



0

t

∞

( e−st f (t − τ)g(τ)dt dτ

∞

( e−s(η+τ) f (η)g(τ)dη dτ

u

0

0

'

∞

=

( ' f (η)e−sη dη

0

=

∞

g(τ)e−sτ dτ

(

0

L [ f ] L [g]

(12.56)

In several applications, the inverse formulation is found to be quite useful, that is, ( ' −1 (12.57) L [ f ] L [g] = f ∗ g L Laplace transform of e−αt , cos (ωt) and sin (ωt). Let α be a constant complex number with a non-negative real part. Then the Laplace transform of f (t) = e−αt is given by  ' (  ∞  ∞ 1 −(s+α)t t=∞ −αt −αt −st −(s+α)t L e e e dt = e dt = − = e  s+α 0 0 t=0

EXAMPLE 12.8.

=

1 s+α

(12.58)

For the special case where α = iω, we have ' ( 1 −iωt L e = s + iω

(12.59)

For cosine and sine, we can use (12.59), Euler’s identity, and the linearity property ( ' ( '   1 1 1 1 = L cos (ωt) L eiωt + e−iωt = + 2 2 s − iω s + iω s = (12.60) s2 + ω2 5

Based on the original definition of convolution given in (12.28), observe that (12.55) is simply the result of restricting f (t) = g(t) = 0 for t < 0, that is,    t  ∞ H (t − τ) f (t − τ) H (τ) g(τ) dτ = f (t − τ)g(τ)dτdτ −∞

0

470

Integral Transform Methods

(

' L

( '   1 1 1 1 iωt −iωt = L e −e − 2i 2i s − iω s + iω ω s2 + ω2

=

sin (ωt)

=

(12.61)

Laplace transform of J ν (t). Using the definition of J ν (x) (cf. (I.43)), the Laplace transform of tν given by (12.43) and the linearity property,

EXAMPLE 12.9.

  L J ν (t)

=

∞  k=0

=

  (−1)k 2k+ν L t k! 22k+ν (ν + k + 1)

  ∞  (−1)k (ν + 2k + 1) 1 2k+ν+1 k! 22k+ν (ν + k + 1) s k=0



=

=

    ∞ (−1)k (ν + 2k + 1) 1 k k! (ν + k + 1) 4s2 k=0 ⎛ ⎤ ⎡ ⎞    k ∞ k−1 k   1 (−1) 1 ⎝1 + ⎣ (k + ν + 1) + j ⎦ ⎠ s(2s)ν k! 4s2

1 s(2s)ν

j =0

k=1

(12.62) where we used the Pockhammer product equation given in (9.11). Note further that in this case, to guarantee that the Pockhammer product will be positive, we need to require that ν > −1. Next, define g(q) as ! ν   4q + 1 − 1 1 ! g(q) = (12.63) 2q 4q + 1 ! or in terms of z = 4q + 1,  ν   2 1 g(z(q)) = z+ 1 z Then

 g(q)q=0  dg  dq q=0

= =

 g(z(q))z=1 = 1       dz  dg −(ν + 1)z − 1  ν+1 =2 = −(ν + 2) dz dq z=1 (z + 1)ν+1 z3 z=1

.. . 

 dk g  dqk q=0

=

(−1)k

k−1 

(k + ν + 1) + j

j =0

which means the Taylor expansion of g(q) around q = 0 is given by ⎤ ⎡ ∞ k−1  (−1)k ⎣ g(q) = 1 + (k + ν + 1) + j ⎦ qk k! k=1

j =0

12.4 Laplace Transforms

471

When we let q = 1/(4s2 ) in (12.63), then (12.62) reduces to ν     !   1 1 + (1/s2 ) − 1 1 ! L J ν (t) = s(2s)ν 1/(2s2 ) 1 + (1/s2 ) √ =

s2 + 1 − s √ s2 + 1

ν (12.64)

Note that this result is valid for ν > −1, including non-integer values.

Laplace transform of Iν (t). To determine the Laplace transform of Iν (t), we can use the definition given in (I.63)   νπi Iν (t) = exp − J ν (it) = (−i)ν J ν (it) 2

EXAMPLE 12.10.

Using the scaling property, (12.51), with a = i and the Laplace transform of J ν (t) given in (12.64), ⎡  ν ⎤ √ 2     −s + 1 − (s/i) ⎥ ⎢1 ⎥ = (−i)ν L J ν (it) = (−i)ν ⎢ L Iν (t) √ ⎣ i ⎦ −s2 + 1  =

EXAMPLE 12.11.

ν √ s − s2 − 1 √ s2 − 1

(12.65)

Laplace transform of H (t) and δ(t).

  L H (t)

 = 0

∞

H (t) e−st dt = −

1 −st t=∞ 1 e t=0 = s s

(12.66)

Using this result plus the fact that δ(t) is defined as the derivative of H (t), ( '     d (12.67) L δ(t) = L H (t) = sL H (t) − H (0) = 1 dt

A summary of the Laplace transforms of some basic functions is given in Table 12.4. Note that the table entries could be extended by use of the properties of Laplace transforms, for example, through the combination of scaling, derivation, integration, and convolution properties. A more complete table is available in the literature.

472

Integral Transform Methods Table 12.4. Laplace transforms of some basic functions f (t) +

0 1

L [f (t)]

Remarks

1 s

See Example 12.11

if t ≤ 0 if t > 0

1

H (t) =

2

δ(t)

1

See Example 12.11

3

e−t

1 s+1

See Example 12.8

4

sin(t)

1 s2 + 1

See Example 12.8

5

cos (t)

s s2 + 1

See Example 12.8

6

tν

(ν + 1) sν+1

See Example 12.5

7

erfc

8

1 √ e−1/(4t) t

9

1 √ e−1/(4t) t3



10

1 √ 2 t



1 −√s e s 7 π −√s e s

See Example 12.6 Left as Exercise E12.8

√ √ 4π e− s

Left as Exercise E12.8

√ ν s2 + 1 − s √ s2 + 1

J ν (t)

 11

Iν (t)

12

 √ tν/2 J ν 2 t

ν √ s − s2 − 1 √ s2 − 1

e−1/s s1+ν

See Example 12.9

See Example 12.10

Left as Exercise E12.9

12.4.3 Decomposition to Partial Fractions For the special case in which the function  f (s) is a rational function of polynomials in s given by N(s)  f (s) = D(s)

(12.68)

where N(s)

=

n−1 

αq sq

(12.69)

q=1

D(s)

=

m  k=1

(s − rk )Nk

with

m  k=1

Nk = n

(12.70)

12.4 Laplace Transforms

473

 f (s) can be separated into n additive terms as follows:

 f (s) =

m 

N k 

k=1

=1

Ak (s − rk )

 (12.71)

whose inverse Laplace transform, after using (12.43) together with the linearity, scaling, and shifting properties, is given by

f (t) =

m 

N k 

k=1

=1

Ak ( − 1)!t−1

 erk t

(12.72)

To determine AKL (with 1 ≤ K ≤ m and 1 ≤ L ≤ NK ), first multiply both sides of (12.71) by (s − rK )NK , f (s) = (K) + (s − rK )NK 

NK 

AK (s − rK )NK −

(12.73)

=1

where,

(K) =

N k   k=K

=1

Ak (s − rK )NK (s − rk )

 (12.74)

Taking the (NK − L)th derivative of (12.73) with respect to s, ' ( L d(NK −L) (K)  (NK − )! d(NK −L) NK  + AK (s − rK )L− (s − rK ) f (s) = (L − )! ds(NK −L) ds(NK −L) =1 (12.75) Finally, because

lim

s→rK

d(NK −L) (K) =0 ds(NK −L)

the limit of both sides of (12.75) as s → rK will yield ⎛

AKL = lim

s→rK

(NK −L)

1 ⎜d ⎝ (NK − L)! ds(NK −L)

⎞ ' ( ⎟ f (s) ⎠ (s − rK )NK 

(12.76)

474

Integral Transform Methods EXAMPLE 12.12.

Consider the function 2s2 + s + 1  f (s) =   s + 21 (s + 2)2 (s + 1 − i)2 (s + 1 + i)2

Using (12.76) we obtain k

rk

Nk

Ak

1 2

−1/2 −2

1 2

3

−1 + i

2

4

−1 − i

2

A11 A21 A22 A31 A32 A41 A42

= −64/225 = −19/18 = −5/6 = (67/100) − (31/100)i = −(7/20) − (1/5)i = (67/100) + (31/100)i = −(7/20) + (1/5)i

After using Euler’s identities for sin(t) and cos(t), (12.72) can be rearranged to be   64 −t/2 5 19 −2t f (t) = − e − t+ e 225 6 18 8 9    2 31 7 67 + e−t t+ sin(t) + − t + cos(t) 5 50 10 50

12.5 Solution of PDEs Using Laplace Transforms The approach of using Laplace transforms to solve partial differential equation follows closely with the Fourier transforms approach. In contrast to using variables such as −∞ < x < ∞, the Laplace transforms need to be applied on semi-infinite variables such 0 ≤ t ≤ ∞. Fortunately, the time variable has this feature, which is mainly why Laplace transforms are usually applied to the time domain. In applications in which one of the variables are defined in a semi-infinite region, such as z ≥ a, the Laplace transform can also be applied on the translated variable t = z − a. The other independent variables may be finite, semifinite, or infinite regions. As it was in Fourier transforms, when taking the Laplace transform with respect to t ≥ 0, the other independent variables are fixed during the transformations. For instance, with u(x, t), one could define a new variable,  s) = L [u(x, t)] U(x,

(12.77)

Some basic rules apply: 1. When taking derivatives with respect to the other independent variables, one can interchange the order of differentiation with the Laplace transform operation, for example, ( ' k dk dk  ∂ u(x, t) = k L [u(x, t)] = k U(x, s) (12.78) L k ∂x dt dt

12.5 Solution of PDEs Using Laplace Transforms

475

2. When taking derivatives with respect to the transformed variable, say t, the derivative property of Laplace transforms can be used, that is, ' L

 ( k−1 n   ∂k k k−n−1 ∂ u  U(x, s) − u(x, t) = s s ∂xk ∂tn t=0

(12.79)

n=0

Note that the initial conditions of the partial derivatives need to be specified by the problem for the successful application of Laplace transforms. The general approach to solving the partial differential equation for u(x, t) can be outlined as follows: 1. Take the Laplace transform of the partial differential equation with respect to  s), with t ≥ 0 and reduce it to an ordinary differential equation in terms of U(x, x as the independent variable. 2. Take the Laplace transform of the conditions at fixed values of x, for example,  t) U(a,

=

d  U(a, t) dx

=

L [u(a, t)] ' ( ∂u L (a, t) ∂x

(12.80) (12.81)

etc.  s). 3. Solve the ordinary differential equation for U(x,  s). 4. Take the inverse Laplace transform of U(x, In several applications, one could use the table of Laplace transforms together with the properties of Laplace transforms. In other cases, one may need to resort to using the method of decomposition to partial fractions and even the theory of residues to evaluate the inverse Laplace transforms.

Laplace transform solution of diffusion equation. Consider the one-dimensional diffusion equation

EXAMPLE 12.13.

∂2u ∂u = (12.82) ∂x2 ∂t with a constant initial condition, u(x, 0) = Ci . Taking the Laplace transform of (12.82) with respect to t, we obtain α2

α2

 d2 U  = Ci − sU dx2

whose general solution is  = Aeλx + Be−λx + Ci U s

(12.83)

√  s /α. The solution will depend on the values of the parameters where λ = and the boundary conditions. Let the boundary conditions be given by u (0, t) = C0

and

lim |u (x, t) | < ∞

x→∞

(12.84)

476

Integral Transform Methods

u(x,t) 100 90 80

Figure 12.3. A plot of the solution given by (12.85).

70 60 1

50 0 x

0.5

10 20

t

0

C0 − Ci . Thus s √  = C0 − Ci e s/α + Ci U s s Based on item 7 in Table 12.4 and the scaling property given in (12.51), '  ( x 1 √ 2 L erfc = e− (x/α) s √ s 2α t Applying these to (12.83), we get A = 0 and B =

The solution is then given by u(x, t) = (C0 − Ci ) erfc



x √ 2α t

 + Ci

(12.85)

A plot of (12.85) with α = 10, C0 = 100, and Ci = 50 is shown in Figure 12.3.

This preceding case applies to the boundary conditions given in (12.84). For other boundary conditions, the use of residues may be needed to obtain the inverse Laplace transform. In Section L.7, we include a few more examples to show how the solutions under different boundary conditions can still be obtained using Laplace transform methods.

12.6 Method of Images For a 2D linear partial differential equation Lu = h(x, y) defined in x > 0 and y > 0 that is subject to two boundary conditions BC1(x, 0) and BC2(0, y) plus the boundedness condition, the Laplace transforms may lack some boundary conditions needed to solve the problem. In this case, the method of Fourier transforms is more appropriate. A technique called the method of images can be used to extend the boundary conditions to allow Fourier transforms to apply. This entails breaking down the problem into two problems uA and uB, in which one of the boundary conditions is made homogeneous. Depending on whether the homogenized boundary condition is Dirichlet or Neumann, the other boundary condition is extended with an odd function or even function, respectively. Thus the original problem Lu = h(x, y)

12.7 Exercises

477

Table 12.5. Boundary conditions of subproblems Original BC

Subproblem A

Subproblem B

u(x, 0) = f (x)

uA(x, 0) = f odd (x)

uB (x, 0) = 0

u(0, y) = g(y)

uA(0, y) = 0

uB (0, y) = g odd (y)

u(x, 0) = f (x)

uA(x, 0) = f even (x)

uB (x, 0) = 0

∂u (0, y) = g(y) ∂x

∂uA (0, y) = 0 ∂x

∂uB (0, y) = g odd (y) ∂x

can be split into two subproblems for LuA = h(x, y) and LuB = h(x, y), and whose boundary conditions are given in Table 12.5, where + + f (x) for x ≥ 0 f (x) for x ≥ 0 even odd (x) = f and f (x) = f (−x) for x < 0 −f (−x) for x < 0 EXAMPLE 12.14.

Consider the Laplace equation ∂2u ∂2u + 2 =0 ∂x2 ∂y

x≥0

∂u (0, y) = 0. Note that using a Laplace transform ∂x with respect to x would require another boundary condition because the reduced ordinary differential equation will be second order. Thus we use the method of images and then apply the Fourier transform method. In this case, there is no need for subproblem B. By extending f to f even and following the same approach in Example 12.4, we obtain    1 y 1 ∞ even y even (x) ∗ u(x, y) = f = f dη (η) 2 2 πx +y π −∞ (x − η)2 + y2   1 ∞ even y 1 ∞ even y = f dη − f dη (η) (−η) 2 2 π 0 (x − η) + y π 0 (x + η)2 + y2    y 1 ∞ y = f (η) − dη π 0 (x − η)2 + y2 (x + η)2 + y2 subject to u(x, 0) = f (x) and

12.7 EXERCISES

E12.1. The Airy equation (9.56) with k = −1 is given by d2 y = xy dx2 whose general solution is given by

(12.86)

y(x) = C1 Ai(x) + C2 Bi (x) where Ai(x) and Bi(x) are the Airy functions of the first and second kind, respectively, defined in (9.58). For y(x) < ∞, we need C2 = 0 because

478

Integral Transform Methods

Bi(x) is unbounded as x → ∞. Using properties of Fourier transforms in Table 12.2, show that dF [y] x y(x) = dω and when applied to (12.86) should yield  3 iω F [y] = C exp 3 Thus we arrive at the conclusion that  3   iω F Ai(x) = exp (12.87) 3 E12.2. Consider the Laplace equation for an infinite domain in two dimensions given by ∂2u ∂2u + 2 =0 ∂x2 ∂y subject to u(x, 0) = f (x) and u(x, y) < ∞ for −∞ ≤ (x, y) ≤ ∞. 1. Applying the Fourier transform on x, show that the solution is given by  1 ∞ y u(x, y) = f (x − τ) 2 dτ π −∞ τ + y2 2. Let τ = y tan θ, show that an alternative form of the solution is given by   1 π/2  u(x, y) = f x − y tan θ dθ π −π/2 3. Plot the solution for the case where f (x) =

2 1 + ex2

E12.3. Consider the third order equation given by ∂u ∂ 3 u + 3 =0 ∂t ∂x subject to u(x, 0) = f (x) and u(x, t) < ∞ for t > 0 and −∞ < x < ∞. Using the Fourier transform approach with respect to x and the transform of Airy function given in (12.87), show that the solution is given by  ∞   1 Ai(ζ) f x − (3t)1/3 ζ dζ u(x, t) = 2π −∞ E12.4. Consider the diffusion equation ∂u ∂2u − μ2 2 = 0 ∂t ∂x subject to u(x, 0) = f (x) and u(x, t) < ∞ for t > 0 and −∞ < x < ∞. Using the Fourier transform with respect to x, show that the solution is given by    ∞ ξ2 1 exp − 2 f (x − ξ) dξ u(x, t) = ! 4μ t 4πμ2 t −∞ However, because the coefficient outside the integral is unbounded when ξ t → 0, let ζ = ! , thus 4μ2 t  ∞   ! 1 2 u(x, t) = √ e−ζ f x − ζ 4μ2 t dζ π −∞

12.7 Exercises

where we assume that f (x) < ∞ as x → ∞. Obtain a time-lapse plot of u(x, t) for −3 ≤ x ≤ 3 and 0 ≤ t ≤ 1 with μ = 1 and ⎧ ⎪ ⎨ 1 for |x| ≤ 1 f (x) = ⎪ ⎩ 0 otherwise E12.5. Recall the Black-Scholes equation described in Exercise E11.9,   ∂u 1 ∂2u ∂u = σ2 x2 2 + r x −u ∂t 2 ∂x ∂x (Note: recall that t is the time from expiry of the option – not actual time.) 1. Taking a hint from the solution of Euler-Cauchy equations (cf. (6.40)), show that with z = ln(x), the Black-Scholes equation can be transformed to be   σ2 ∂ 2 u ∂u σ2 ∂u = + r− − ru ∂t 2 ∂z2 2 ∂z 2. Next, using u(z, t) = e(Az+Bt) q(z, t), find the values of A and B such that the equation reduces to a diffusion equation given by ∂q σ2 ∂ 2 q = ∂t 2 ∂z2 3. Using the solution given in Exercise E12.4, show that the solution that satisfies the “European call option”: u(0, t) = 0, u(x, t) → x as x → ∞ and u(x, 0) = max(x − K, 0), where K is the strike price, is the BlackScholes formula for European call option given by (12.88) xN (d1 ) − Ke−rt N (d2 )   2 r + (σ /2) t + ln(x/K) d1 = √ σ2 t   r − (σ 2 /2) t + ln(x/K) d2 = √ σ2 t where N (·) is the cumulative normal distribution function defined as follows:  √   ξ 1 + erf ξ/ 2 1 2 N (ξ) = √ e−η /2 dη = 2 2π −∞ (Hint: By completing squares in the arguments, it can be shown that  2     ξ   2  1 1 b − 4ac b + 2aξ exp − aη + bη + c dη = √ exp N √ √ 4a π −∞ a 2a where a > 0.) u(x, t)

=

E12.6. Obtain the solution for the reaction-diffusion equation with time-dependent coefficient K(t) given by ∂u ∂2u = D 2 − K(t)u ∂t ∂x subject to u(x, 0) = f (x) and u(x, t) < ∞ for t > 0 and −∞ < x < ∞; use the Fourier transform method with respect to x. Compare the solution with one obtained using the transformation suggested in Exercise E11.11.

479

480

Integral Transform Methods

E12.7. Consider the Laplace equation ∂2u ∂2u + 2 = 0 ; −∞ ≤ x ≤ ∞ ; y ≥ 0 ∂x2 ∂y subject to Neumann condition given by ∂u (x, 0) = g(x) and u(x, y) < ∞ ∂y ∞ where −∞ g(x)dx = 0. Show that the general solution is given by  2   ∞ 1 y + τ2 u(x, y) = g(x − τ) ln dτ 2π −∞ α2 + τ2 y where α is an arbitrary constant. (Hint: Let u = α φ(x, η)dη. Then show that ∇ 2 φ = 0 with φ(x, 0) = g(x). After solving for φ(x, y), obtain u(x, y).) E12.8. Using the definition of Laplace transforms, prove the following identities: 7 '  ( 1 1 π −√s L √ exp − = e (12.89) 4t s t '  ( √ √ 1 1 L √ exp − = 4π e− s (12.90) 4t t3 E12.9. Using a similar approach as Example 12.9, show that  √  e−1/s  L tν/2 J ν 2 t = ν+1 (12.91) s Thus, using the scaling properties, show that for γ > 0 '  γ (  √  1 L−1 exp = I0 2 γx (12.92) s s E12.10. To obtain the inverse Laplace transform of   1 γs F (s) = exp s s+1 First show that       γ 1 ξs 1 1 γs exp dξ + = exp s+1 s+1 s s s+1 0 Using this identity and (12.92), prove that ' (  γ  !  1 γs L−1 exp = H (t) + e(ξ−t) J 0 2 ξt dξ (12.93) s s+1 0 E12.11. The concentration of a reactant A undergoing a first-order reaction A → P in a plug flow reactor can be modeled by ∂c ∂c +v = −kc for 0 ≤ x ≤ L, t ≥ 0 ∂t ∂x subject to c(x, 0) = f (x) and c(0, t) = g(t), where v is the constant velocity of the flow. Use the Laplace transform method to solve this problem. Compare this with the solution obtained using the method of characteristics (cf. Section 10.1). (Note that the method of characteristics can also handle an nth -order reaction, i.e., with −kcn instead of −kc.) E12.12. Consider a heat exchanger tube immersed in a temperature bath. The temperature of fluid flowing through the tube can be modeled as   ∂u ∂u +v = (H) ub(t) − u ∂t ∂x

12.7 Exercises

481

subject to u(x, 0) = f (x) and u(0, t) = g(t) for 0 ≤ x ≤ L and t ≥ 0, where v, H, and ub(t) are velocity, heat transfer coefficient, and the time-varying temperature of the bath, respectively. Solve the problem using the Laplace transform method. E12.13. Consider the system of first-order equations given by ∂u ∂v =v−u and =u−v ∂x ∂t subject to the conditions: u(x, 0) = 0, v(x, 0) = 0 and u(0, t) = u0 , where x ≥ 0 and t ≥ 0. Using Laplace transforms, show that the solution for u is given by '  x  !  ( −(ξ+t) e I0 2 ξt dξ u(x, t) = u0 H (t) − 0

(Hint: You may need equation (12.93).) Obtain a set of time-lapse plots of u(x, t)/u0 . E12.14. The dynamic equations describing the current and voltage distribution along a thick cable can be modeled by ∂I ∂V +C + GV = 0 ∂t ∂t ∂V ∂I + L + RI = 0 ∂t ∂t where I(x, t) and V (x, t) are the current and voltage values, whereas C, R, L, and G are constant capacitance, resistance, inductance, and leakage, respectively. 1. Show that these equations can be combined to yield 2 ∂2u ∂u 2∂ u + + β) + αβu = γ (α ∂t2 ∂t ∂x2 G R 1 where u is either V (x, t) or I(x, t), with α = , β = and γ = . These C L LC equations are known as the telegraph equations. 2. For the special case of β = α, use the Laplace transform method to solve the equation for the conditions ∂u u(0, t) = g(t); u(x, 0) = (x, 0) = 0 and u(∞, t) = 0 ∂t E12.15. Consider the Nusselt problem describing the heat transfer for a plug flow, as given by the following equation:

v ∂u ∂ 2 u 1 ∂u = 2 + α ∂z ∂r r ∂r

0 ≤ r ≤ R,

0≤z≤L

∂u (z, 0) = 0. ∂r 1. Applying Laplace transform with respect to z, show that this results in   √  uin uR − uin J 0 ia s  √ L [u] = + s s J 0 ib s 7 7 √ v v where a = r and b = R (with i = −1). α α 2. Using the approach used in Example 12.7 to find the inverse Laplace transform, obtain the solution u(z, r). subject to: u(z, R) = uR , u(0, r) = uin and

482

Integral Transform Methods

E12.16. Consider the shallow water model given in Example 10.8. After linearization, we obtain the system described as follows: ⎛ ⎞ ⎛ ⎞ ⎛ ∂h ⎞ ⎛ ⎞ ∂h 0 H 0 ⎜ ∂t ⎟ ⎜ ⎜ ⎟ ⎟ ⎜ ∂x ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟⎜ ⎟ ⎜ ⎟+⎝ ⎟=⎜ ⎠ ⎠ ⎝ ⎝ ∂v ⎠ ⎝ ∂v ⎠ g 0 0 ∂t ∂x where h(x, t) and v(x, t) are the height of the surface and velocity along x, respectively. The constants g and H are the gravitational acceleration and mean depth, respectively. Assume that the initial and boundary conditions are h(x, 0) = v(x, 0) = 0 and v(0, t) = f (t), with v(x, t) < ∞. Using the Laplace transform method, show that the solution is given by 6     x H x v(x, t) = f t − √ and h(x, t) = f t− √ g Hg Hg E12.17. Consider the diffusion equation given by ∂u ∂2u =D 2 ∂t ∂x subject to u(x, 0) = f (x) and u(0, t) = g(t) for x ≥ 0 and t ≥ 0. Based on the discussion in Section 12.6, the problem can be split into two subproblems: ∂uA ∂ 2 uA =D 2 ∂t ∂x

subject to uA(x, 0) = f odd (x), uA(0, t) = 0

∂uB ∂ 2 uB =D 2 ∂t ∂x

subject to uB(x, 0) = 0, uB(0, t) = g(t)

where f odd is the odd function extension of f (x). Solve for uA using the Fourier transform method and uB using the Laplace transform method. (Note that because the time derivative is first order, only one initial condition is needed.) Thus obtain the final solution as u(x, t) = uA + uB.

13

Finite Difference Methods

In this chapter, we discuss one powerful approach to obtain a numerical solution of partial differential equations. The basic approach is to replace derivatives by discrete formulas called finite difference approximations. After these approximations are applied to the given differential equations, the boundary conditions are included by modifying the equations that involve the boundary points. This often results in a large and sparse matrix equation, in which the desired values at the chosen grid points are combined as a vector or a matrix, depending on whether the problem involves one dimension or two dimensions. For steady-state cases, the unknown vector can be obtained by solving an algebraic equation. Conversely, for non–steady-state cases, algebraic iteration is used to obtain a time-marching solution. Throughout the chapter, we limit our discussion to a discretization based on uniform grids. Under this assumption, different finite difference approximations can be formulated using the Taylor series expansions as discussed in Section 13.1. Several formulations are possible depending on the choice of neighboring points for different order of derivatives. Formulas for various approximations of first-order and second-order derivatives, including their order of accuracy, are given in Tables 13.1 and 13.2, respectively. Once the derivatives in the differential equations are replaced with their finite difference approximations, the resulting formulas can be recast as matrix algebraic equations. We limit our applications to second-order linear partial differential equations. We first discuss the time-independent cases before moving on to timedependent cases. For the time-independent cases, we start with the solutions of one-dimensional cases. Although this is technically an ordinary differential equation, it is an appropriate point to develop the basic formulas for handling either Dirchlet, Neumann, or Robin conditions. Several of the matrices are used when extending the method to two and three dimensions. For the 2D and 3D cases, additional matrix operations such as Kronecker products, vectorization, and Haddamard products (cf. Table 1.3), including their properties (cf. Table 1.4), are used extensively in our formulas. Although the finite difference method is used most often in rectangular domains, the case of cylindrical or spherical coordinates can be handled with minimal modifications, as discussed in Section 13.2.3. Ultimately, the matrix equations reduce to the linear equation of the form Ax = b. Thus the techniques covered in Chapter 2 can be implemented to take advantage of the sparseness and structure of the equations. 483

484

Finite Difference Methods

Having developed matrix equations for one, two, or three-dimensional equations, the transition to time-dependent cases is straightforward. This is achieved by using an approach called the semi-discrete method, which essentially reduces the partial differential equations first into initial value problems. This means that the techniques covered in Chapter 7 can be used directly. However, due to size of the problems, the Euler methods (forward and backward difference), as well as the averaged version known as Crank Nicholson methods, are often the methods of choice. As it was in Chapter 7, the issue of stability becomes important, where implicit methods have the advantages of much larger stability regions. In Section 13.4, we discuss the stability analysis based on the spectral radius, but we also include another method based on Fourier analysis known as the von Neumann analysis. We include the use of finite difference equations for handing hyperbolic equations in Section M.3 as an appendix. Recall that the challenge for handling these types of equations is that the solutions are expected to travel as waves. Thus discontinuities will be propagated in the path of characteristics (in nonlinear cases, these could also involve shock formations). Some of the methods are simple substitution of finite difference formulas such as the upwind formulas. However, other methods, such as the Lax-Wendroff method, which uses the Taylor series approximation directly on the differential equation, will arrive at a different scheme altogether. We do not cover the solution of nonlinear partial differential equations. Instead, we just note that the extension to some nonlinear problems, such as for semilinear or quasilinear partial differential equations, are often straightforward. The additional complexity will be the convergence of the linearized formulas, either by successive substitution or Newton-type approaches.

13.1 Finite Difference Approximations As mentioned in the introduction, the main steps in a finite difference scheme involve the following steps: 1. 2. 3. 4.

Discretize the domain. Approximate the derivatives. Include the boundary conditions. Solve the algebraic equations.

In this section, we discuss the first and second steps in more detail, that is, how to obtain various finite difference approximations of derivatives. Specifically, we develop the formulas for first- and second-order derivatives. The third step, that is, the inclusion of boundary conditions, is discussed in the next section. Let us first set some specific index notations. We will assume that t, x, y, and z domains have been discretized by uniform increments of t, x, y, and z, (q) respectively. Thus let uk,n,m be defined as (q)

uk,n,m = u (q t, k x, n y, m z)

(13.1)

Throughout this chapter, we reserve the integers q, k, n, and m as indices for t, x, y, and z, respectively. We also assume that each spatial dimension has already been scaled to be in the range of [0, 1]. For instance, the range of discretized x will be (0, x, . . . , K x,

13.1 Finite Difference Approximations

485

(K + 1) x), where x = 1/(K + 1). The points x = 0 and x = (K + 1) x are at the boundaries. When some of the dimensions are fixed, the corresponding superscripts or subscripts will be suppressed. For instance, in a time-varying system that is dependent (q) on only one spatial dimension, we use uk = u (q t, k x), whereas for a timeindependent system in two spatial dimensions, we have uk,n = u (k x, n y). Let us now start with the approximation of a first-order derivative. The Taylor series expansion of uk+1 = u (x + x) around x = xk is given by   du  1 d2 u  uk+1 = uk +

x +

x2 + . . . (13.2) dx xk 2 dx2 xk Rearranging,

 du  uk+1 − uk = + O ( x) dx xk

x

then dropping the O ( x) terms,

 du  uk+1 − uk ≈ dx xk

x

(13.3)

which is a first-order approximation known as the forward difference approximation. We could also expand uk−1 = u(x − x, t) around x = xk ,   du  1 d2 u  uk−1 = uk −

x +

x2 + . . . (13.4) dx xk 2 dx2 xk Rearranging,

 du  uk − uk−1 = + O ( x) dx xk

x

then dropping the O ( x) terms,

 du  uk − uk−1 ≈ dx xk

x

(13.5)

which is a first-order approximation known as the backward difference approximation. Alternatively, we can subtract (13.4) from (13.2),   du 

x3 d3 u  uk+1 − uk−1 = 2

x + 2 + ... (13.6) dx xk 6 dx3 xk from which we get

   du  uk+1 − uk−1 = + O x2  dx xk 2 x  2 and upon dropping the O x terms,  du  uk+1 − uk−1 ≈  dx xk 2 x

(13.7)

(13.8)

This is a second-order approximation known as the central difference approximation.

486

Finite Difference Methods

The finite difference approximations of higher order derivatives can likewise be obtained as a linear combination of the neighboring points as follows:  2 dP u  1  ≈ D = α j uk+j P dxP xk

xP j =

(13.9)

1

The choice for the indices 1 and 2 will determine both the order of approximation as well as the bandwidth of the matrices used to solve the finite difference equations once they are applied to the differential equations. For instance, the forward, backward, and central difference formula given in (13.3), (13.5), and (13.7) uses the limits (1 , 2 ) = (0, 1), (1 , 2 ) = (−1, 0), and (1 , 2 ) = (−1, 1), respectively. Both forward and backward approximation formulas have bandwidths of 2 yielding a first-order approximation, whereas the central approximation formula has a bandwidth of 3 but yields a second-order approximation. One approach to find the coefficients α j is known as method of undetermined coefficients and is based on the Taylor series of u or of its Mth derivative at x = xk+j , which we can rewrite in the following form:  ∞ dM u  1  = ϕι γ(ι−M),j dxM xk+j

xM

(13.10)

ι=0

where  d u  ϕr = xr , dxr xk r

γr,s

⎧ r s /r! ⎪ ⎪ ⎪ ⎨ 1 = ⎪ ⎪ ⎪ ⎩ 0

if r > 0 if r = 0

(13.11)

if r < 0

The form given in (13.9) does not include derivatives at neighboring points. Thus substituting (13.10) with M = 0, 1 ≤ 0, 2 ≥ 0 and P ≤ (2 − 1 ) into (13.9),  dP u  dxP xk ϕP

xP

=

=

2  1 αu P j k+j

x j =1 ⎡ ⎤ ⎡ (2 −1 ) 2  ϕι  ⎣ γιj α j ⎦ + ⎣

xP j = ι=0

∞ 

ν=(2 −1 )+1

1

⎤ 2 ϕν  γνj α j ⎦

xP j = 1

By setting the second sum on the right side to be the truncation error, we have the following lemma: The coefficients α j of the finite difference approximation of a Pth -order derivative under the form given by (13.9), with 1 ≤ 0, 2 ≥ 0 and P ≤ (2 − 1 ) is given by

LEMMA 13.1.

⎞ ⎛ γ0,1 α1 ⎜ .. ⎟ ⎜ .. ⎝ . ⎠=⎝ . α2 γ2 −1 ,1 ⎛

··· .. . ···

γ0,2 .. . γ2 −1 ,2

⎞−1 ⎟ ⎠

eP+1

(13.12)

13.1 Finite Difference Approximations

487

where eP+1 is the (P + 1)th unit vector of length (2 − 1 + 1), yielding a truncation error given by ⎡

∞ 

Error =

⎣

ν=(2 −1 )+1

2 

⎤ γνj α j ⎦

j =1

  dν u 

xν−P dxν xk

(13.13)

EXAMPLE 13.1. Approximation of Second-Order Derivatives. For the finite difference approximation of the second-order partial derivative with respect to x, let P = 2, 1 = −1 and 2 = 1. Then (13.12) becomes ⎞ ⎛ ⎞ ⎛ ⎞−1 ⎛ ⎞ ⎛ 1 1 1 1 0 α−1 ⎝ α0 ⎠ = ⎝ −1 0 1 ⎠ ⎝ 0 ⎠ = ⎝ −2 ⎠ α1 1 1/2 0 1/2 1

Thus

 d2 u  1 ≈ (uk−1 − 2uk + uk+1 ) dx2 xk

x2

(13.14)

Looking at the truncation error, we see that the leading term in the summation found in (13.13) is zero, that is, (−1)3 13 −0·2+ =0 3! 3! This means that the level of approximation in (13.14) is second-order, which is a fortunate case because the leading terms of the truncation errors do not vanish in general. γ3,−1 α−1 + γ3,0 α0 + γ3,1 α1 =

The finite difference form given in (13.9) can be extended to include derivatives at the neighboring points:  dP u  ≈ Dp = dxP xk

 j ∈[1 ,2], λ∈[0,P]



1

xP−λ



 α j,λ

 dλ u  dxλ xk+j

(13.15)

One reason for including derivatives of neighboring points is to conveniently apply Neumann and Robin boundary conditions at the points near the boundary. Another reason is to obtain higher order accuracy while balancing smaller matrix bandwidths against increased complexity. By applying the general Taylor series given by (13.10) to (13.15), we have the following generalization of Lemma 13.1, which we now refer to as the finite difference approximation lemma The (Q + 1) coefficients α j,λ for the finite difference approximation (13.15) can be obtained by solving the following set of simultaneous equations:

LEMMA 13.2.

 j,λ

γ(ι−λ),j α j,λ = δPι

;

j ∈ [1 , 2 ], λ ∈ [0, P], ι = 0, . . . , Q

(13.16)

488

Finite Difference Methods

where δrs is the Kronecker delta (i.e., δrr = 1 and 0 otherwise). The truncation error is given by ⎡ ⎤  ν  ∞   d u  ⎣ γ(ν−λ),j α j,λ ⎦ Error =

xν−P (13.17) dxν xk ν=Q+1

j ∈[1 ,2 ],λ∈[0,P]1

Let J and  be the list of indices for j and λ, respectively. For instance, set J = {−1, 0, 1} and  = {1, 0, 0} for the approximation given by    d2 u 1 du  ≈ α−1,1 x + α0,0 uk + α1,0 uk+1 dx2

x2 dx xk−1

EXAMPLE 13.2.

Lemma 13.2 yields the following equation: ⎛ ⎞⎛ ⎞ ⎛ ⎞ 0 1 1 α−1,1 0 ⎝ 1 0 1 ⎠ ⎝ α0,0 ⎠ = ⎝ 0 ⎠ −1 0 1/2 α−1,0 1 or d2 u 2 ≈ 2 dx 3 x2



⎞ ⎛ ⎞ α−1,1 −1 ⎝ α0,0 ⎠ = 2 ⎝ −1 ⎠ 3 α−1,0 1 ⎛

→

  du  − x − uk + uk+1 dx xk−1

(13.18)

Furthermore, the lemma also predicts Error = O ( x) for this approximation.

Remarks: A MATLAB code to obtain the coefficients of a finite difference approximation of a Pth -order derivative based on a given list of indices J and  is available on the book’s webpage as eval_pade_gen_coef.m. The function is invoked by the statement [v,ord]=eval_pade_gen_coef(J,Lambda,P) to obtain v as the vector of coefficients and ord as the order of approximation. A list of common finite difference approximations for the first- and secondorder derivatives based on the method of undetermined coefficients is given in Tables 13.1 and 13.2, respectively. These are used later in formulating the numerical solution of differential equations. As expected, more terms are needed for increased precision. A balance has to be struck between a smaller x or using more terms in the finite difference approximations. Having more terms in the approximation formula will increase the number of computations due to a matrix with larger bandwidths. However, a smaller x will also increase the number of computations because the size of the matrix will have to be enlarged to achieve a similar precision. Note that items 7 through 12 of Table 13.1 and items 5 through 12 of Table 13.2 involve derivatives at the leftmost or rightmost neighbor. These approximation formulas are used when dealing with Neumann boundary conditions. For mixed derivatives, the same method of undetermined coefficients can be used with the Taylor series expansion of multivariable functions but would require ∂2u , we have much more complex formulation. For the case of ∂t∂x     ∂ 2 u  1  (q+1) (q+1) (q−1) (q−1) u ≈ − u (13.19) − u − u k+1 k−1 k+1 k−1 ∂x∂t (xk ,tq ) 4 x t

13.1 Finite Difference Approximations  Table 13.1. Finite difference approximations of first-order derivatives,

Item

489

  du  dx xk

Approximation formula

Error order

1

 1  −uk + uk+1

x

O ( x)

2

 1  −uk−1 + uk

x

O ( x)

3

 1  −uk−1 + uk+1 2 x

  O x2

4

 1  +uk−2 − 8uk−1 + 8uk+1 − uk+2 12 x

  O x4

5

 1  −3uk−1 − 10uk + 18uk+1 − 6uk+2 + uk+3 12 x

  O x4

6

 1  −uk−3 + 6uk−2 − 18uk−1 + 10uk + 3uk+1 12 x

  O x4

7

1 3

8

'   ( 1 2 2 du − uk−1 + uk + 3

x

x dx k+1

  O x2

9

'   ( 1 du 38 9 54 7 −3 − uk−1 − uk + uk+1 − uk+2 75 dx k−2 x

x

x

x

  O x4

10

'   ( 1 du 197 279 99 17 18 − uk + uk+1 − uk+2 + uk+3 150 dx k−1

x

x

x

x

  O x4

11

'   ( 1 7 54 9 du 38 uk−2 − uk−1 + uk + uk+1 − 3 75 x

x

x

x dx k+2

  O x4

12

'   ( 1 17 99 279 du 197 − uk−3 + uk−2 − uk−1 + uk + 18 150

x

x

x

x dx k+1

  O x4

'

du dx



2 2 − uk + uk+1

x

x k−1

(

  O x2

  with Error = O t2 , x2 . Likewise, we have      ∂ 2 u  1 u ≈ − −u − u + u k+1,n+1 k+1,n−1 k−1,n+1 k−1,n−1 ∂x∂y (xk ,yn ) 4 x y

(13.20)

  with Error = O x2 , y2 . The details for this result can be found in example M.1 contained in the appendix as Section M.1. For higher order approximations, a simpler

490

Finite Difference Methods  Table 13.2. Finite difference approximations of second order derivatives,

  d2 u  dx2 xk

Item

Approximation formula

Error order

1

 1  uk−1 − 2uk + uk+1 2

x

  O x2

2

 1  −u + 16u − 30u + 16u − u k−2 k−1 k k+1 k+2 12 x2

  O x4

3

 1  10u − 15u − 4u + 14u − 6u + u k−1 k k+1 k+2 k+3 k+4 12 x2

  O x4

4

 1  uk−4 − 6uk−3 + 14uk−2 − 4uk−1 − 15uk + 10uk+1 2 12 x

  O x4

5

'   ( 1 du −2 x − 2u + 2u k k+1 3 x2 dx k−1

O ( x)

6

  ( ' 1 du − 2u + 2 x 2u k−1 k 3 x2 dx k+1

O ( x)

7

'   ( 1 du −6 x − 4uk + 2uk+1 + 2uk+2 11 x2 dx k−1

  O x2

8

'   ( 1 du 2u + 2u − 4u + 6 x k−2 k−1 k 11 x2 dx k+1

  O x2

9

  ⎡ ⎤ du 1 30 x + 946uk−1 − 1905uk ⎣ ⎦ dx k−2 822 x2 + 996uk+1 − 31uk+2 − 6uk+3

  O x4

10

  ⎡ ⎤ du 1 −600 x + 945u − 3548u k k+1 ⎣ ⎦ dx k−1 1644 x2 + 3918uk+2 − 1572uk+3 + 257uk+4

  O x4

11

⎡ ⎤ −6uk−3 − 31uk−2 + 996uk−1 − 1905u k  1 ⎣ ⎦ du +946uk+1 − 30 x 822 x2 dx k+2

  O x4

12

⎡ ⎤ +257uk−4 − 1572uk−3 + 3918uk−2 −  3548uk−1 1 ⎣ ⎦ du + 945uk + 600 x 1644 x2 dx k+1

  O x4

but slightly limited approach for mixed partial derivatives is to build up the finite difference approximation by using the fact that ∂2u ∂ = ∂x∂y ∂x



∂u ∂y



∂3u ∂ , = ∂x∂ 2 y ∂x



∂2u ∂y2

 , . . . , etc.

13.2 Time-Independent Equations

491

Thus (13.20) could also have been derived by applying central difference approximations on the variable y followed by another central difference approximation on the variable x, that is,        ui,n+1 − ui,n−1  ∂2u ui,n+1 − ui,n−1  1 = − + O x2 , y2   ∂x∂y 2 y 2 y 2 x i=k+1 i=k−1 In case one needs to accommodate a Neumann boundary condition at y = 0, another useful form based on item 7 in Table 13.1 is given by 8     ∂2u 1

y ∂u  ∂u  2 = − − (uk+1,n − uk−1,n )   ∂x∂y 2 x y 3 ∂y k+1,n−1 ∂y k−1,n−1 3 ( 2 + (uk+1,n+1 − uk−1,n+1 ) (13.21) 3 The derivation of (13.21) is included as exercise E13.2.

13.2 Time-Independent Equations We first apply the finite-difference approach to solve partial differential equations that are independent of time. The finite difference approximations are expressed in matrix forms. This reduces the problem to the solution of a set of simultaneous equations for u at the grid points.

13.2.1 One-Dimensional Case A general second-order linear differential equation is given by d2 u du + β(x) + ζ(x)u + η(x) = 0 dx2 dx

(13.22)

Note that the problem is a two-point boundary value problem that can also be approached using the techniques given in Section 7.5 based on shooting methods. Nonetheless, the matrix formulations in this section are used when they are extended to the 2D and 3D cases. Define the following vectors and matrices as: ⎞ u1 ⎟ ⎜ u = ⎝ ... ⎠ uK ⎛

⎛ ⎜ β=⎝

β1

0 ..

0

.

⎞

⎛

⎟ ⎠

⎜ ζ=⎝

βK

ζ1

0 ..

0

. ζK

⎞ ⎟ ⎠

⎞ η1 ⎟ ⎜ η = ⎝ ... ⎠ ηK ⎛

where uk = u(k x), βk = β(k x), ζk = ζ(k x), and ηk = η(k x). The terms connected with the derivatives are given in matrix form as du → D 1 u + b1 dx

and

d2 u → D 2 u + b2 dx2

The elements and structure of the K × K matrix DP are formed using the formulas in Tables 13.1 and 13.2, depending on the choice of precision and types of boundary conditions, whereas the elements of the K × 1 vector bP contain the given boundary

492

Finite Difference Methods

values of the problem. Thus the finite difference approximation of (13.22) can be represented by a matrix equation (D2 u + b2 ) + β (D1 u + b1 ) + ζu + η = 0 where A = D2 + βD1 + ζ

and

→

Au = b

(13.23)

  b = − b 2 + β b1 + η

and the solution can be found using techniques found in Chapter 2 that take advantage of the band structure and sparsity of matrix A. What remains at this point is to specify D1 , D2 , b1 , and b2 . We limit our discussion to two cases. In the first case, the boundary conditions are Dirichlet type at both x = 0 and x = 1, that is, u(0) = Di0 and u(1) = Di1 . Using the formulas in Tables 13.1 and 13.2, we have for Error = O x2 (where we denote the Dirichlet or Neumann boundary condition by (Di) or (Neu), respectively as a superscript for x = 0 and as a subscript for x = 1) ⎞ ⎛ ⎞ ⎛ −Di0 0 1 ⎜ 0 ⎟ ⎟ ⎜ ' ((Di)  ' ((Di)  .. .. ⎟ ⎟ . . 1 ⎜ 1 ⎜ −1   ⎟ ⎜ .. ⎟ ⎜ and b = D1 =   ⎟ ⎜ 1 . ⎟ ⎜ . .   ⎟ ⎜ 2 x 2 x . . (Di) O( x2 ) (Di) O( x2 ) ⎝ . 1 ⎠ . ⎝ 0 ⎠ −1 0 Di1 (13.24) ⎛

' ((Di)   D2  (Di) 

1 =

x2

O( x2 )

−2

⎜ ⎜ 1 ⎜ ⎜ ⎝

..

..

..

.

⎛

⎞

1 .. .

.

. 1

⎟ ' ((Di)  ⎟ 1  ⎟ and b2 =  ⎟ 2 (Di) O( x2 ) x 1 ⎠ −2

⎜ ⎜ ⎜ ⎜ ⎜ ⎝

Di0 0 .. . 0 Di1

⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠

(13.25)   For higher precision, that is, Error = O x4 , we have ⎛ −10 18 −6 1 ⎜ −8 0 8 −1 ⎜ ⎜ 1 −8 0 8 −1 ' ((Di)  1 ⎜  ⎜ . . . .. .. .. .. D1 =  ⎜ . 12 x ⎜ (Di) O( x4 ) ⎜ 1 −8 0 ⎜ ⎝ 1 −8 −1

⎛

' ((Di)   b1  (Di) 

O( x4 )

−3Di0 Di0 0 .. .

⎜ ⎜ ⎜ 1 ⎜ ⎜ = ⎜ 12 x ⎜ ⎜ 0 ⎜ ⎝ −Di1 3Di1

6

⎞

..

. 8 0 −18

⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ −1 ⎟ ⎟ 8 ⎠ 10

⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠

(13.26)

13.2 Time-Independent Equations

493

−13

0.6

0

0.5

−0.5

0.4

x 10

−1

u

Error 0.3

−1.5

0.2

−2

0.1 0

0.2

0.4

0.6

0.8

−2.5 0

1

0.2

0.4

x

0.6

0.8

1

x

Figure 13.1. A plot of the exact solution for u (solid line) and finite difference solution (points). On the right is a plot of the errors of the approximate solution from the exact solution.

⎛

'

((Di)   D2  (Di) 

O( x4 )

⎜ ⎜ ⎜ 1 ⎜ ⎜ = ⎜ 12 x2 ⎜ ⎜ ⎜ ⎝

−15 16 −1

−4 −30 16 .. .

14 16 −30 .. .

−6 −1 16 .. .

−1 .. .

−1

16 −1 −6

−30 16 14

1 ⎛

' ((Di)   b2  (Di) 

EXAMPLE 13.3.

O( x4 )

10Di0 −Di0 0 .. .

⎜ ⎜ ⎜ 1 ⎜ ⎜ = ⎜ 12 x2 ⎜ ⎜ 0 ⎜ ⎝ −Di1 10Di1

⎞

1

..

. 16 −30 −4

⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ −1 ⎟ ⎟ 16 ⎠ −15

⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠

(13.27)

Consider the differential equation

d2 u du + (3 − x) + 5u = 10x3 + 22.5x2 − 3x − 5.5 2 dx dx subject to Dirichlet conditions u(0) = 0.1 and u(1) = 0.6. The exact solution is known to be: u(x) = 5x3 − 7.5x2 + 3x + 0.1. Using x = 0.01 plus the matrices defined in (13.26) and (13.27) for high precision, we obtain the plots for the solution and errors shown in Figure 13.1. Note that the errors are within ±10−12 . It can be shown that using (13.24) and (13.25) instead would have yielded errors within ±10−3 .

For the second case, we consider the boundary condition at x = 0 to be a Neumann type and the boundary condition at x = 1 to remain a Dirichlet type, that is,

494

Finite Difference Methods

du/dx(0) = Neu0 and u(1) = Di1 . Then we can use item 7 in Table 13.1 and item 7 in Table 13.2 for Error = O x2 , ⎛

'

((Neu)   D1  (Di) 

− 34 ⎜ −1 1 ⎜ ⎜ = ⎜ 2 x ⎜ ⎝

O( x2 )

0 .. .

O( x2 )

⎜ 1 ⎜ ⎜ = ⎜ 2 x ⎜ ⎝ ⎛

'

((Neu)   D2  (Di) 

O( x2 )

1 =

x2

−4 11

⎜ 1 ⎜ ⎜ ⎜ ⎜ ⎝

O( x2 )

1 =

x2

2

x 3

−2 .. .

. 0 −1

⎜ ⎜ ⎜ ⎜ ⎜ ⎝

Neu0 0 .. .

⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠

−6

x 11

..

. −2 1

Neu0

0 .. . 0 Di1

(13.28)

⎞

2 11

1 .. . 1

⎟ ⎟ ⎟ ⎟ ⎟ 1 ⎠ 0

..

0 Di1 2 11

⎛

' ((Neu)   b2  (Di) 

1 .. . −1

⎛

' ((Neu)   b1  (Di) 

⎞

4 3

⎟ ⎟ ⎟ ⎟ ⎟ 1 ⎠ −2

⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠

(13.29)

  The matrices for the second case above corresponding to Error = O x4 is left as an exercise in E13.3. Another approach to handling Neumann boundary conditions is to apply a central difference approximation of the derivative itself. For instance, let the Neumann boundary condition be given as du/dx(0) = Neu0 . Then this can be approximated first as  du  −u−1 + u1 ≈ −→ u−1 = u1 + 2 xNeu0 (13.30)  dx x0 2 x Because u−1 is a value at a fictitious or ghost point; that is, it is not in the actual problem domain, the approach is sometimes known as method of ghost points. To complete this method, the solution domain will be extended to include the point at x0 ; that is, we extend the previous definitions as ⎛ ⎜ ⎜ u∗ = ⎜ ⎝

u0 u1 .. . uK

⎞

⎛

⎟ ⎜ ⎟ ∗ ⎜ ⎟ β =⎜ ⎠ ⎝

β0

⎞ β1 ..

0

⎛

ζ0

⎜ 0 ⎟ ζ1 0 ⎟ ∗ ⎜ ⎟ ζ =⎜ . .. ⎠ ⎝ . βK 0 ζK

⎞

⎛

⎟ ⎜ ⎟ ∗ ⎜ ⎟ η =⎜ ⎠ ⎝

η0 η1 .. . ηK

⎞ ⎟ ⎟ ⎟ ⎠

13.2 Time-Independent Equations

495

The matrix formulation becomes ⎛

' ((Neu)   D1∗  (Di) 

O( x2 )

0 ⎜ −1 1 ⎜ ⎜ = ⎜ 2 x ⎜ ⎝

0 0 .. .

⎛

' ((Neu)   b∗1  (Di) 

O( x2 )

⎜ 1 ⎜ ⎜ = ⎜ 2 x ⎜ ⎝ ⎛

' ((Neu)   D2∗  (Di) 

O( x2 )

−2 ⎜ 1 1 ⎜ ⎜ = ⎜

x2 ⎜ ⎝

' ((Neu)   b∗2  (Di) 

O( x2 )

1 =

x2

1 .. . −1

..

. 0 −1

⎟ ⎟ ⎟ ⎟ ⎟ ⎠

0 Di1

(13.31)

⎞ 1 .. . 1

..

. −2 1

−2 xNeu0 0 .. .

⎜ ⎜ ⎜ ⎜ ⎜ ⎝

⎟ ⎟ ⎟ ⎟ ⎟ 1 ⎠ 0

⎞

2 xNeu0 0 .. .

2 −2 .. .

⎛

⎞

0 Di1

⎟ ⎟ ⎟ ⎟ ⎟ 1 ⎠ −2

⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠

(13.32)

and the solution can be obtained by solving A∗ u∗ = b∗

(13.33)

where A∗ = D2∗ + β∗ D1∗ + ζ∗

EXAMPLE 13.4.

and

  b∗ = − b∗2 + β∗ b∗1 + η∗

Consider again differential equation of Example 13.3

d2 u du + (3 − x) + 5u = 10x3 + 22.5x2 − 3x − 5.5 dx2 dx except this time we have a Neumann condition du/dx(0) = 3 and a Dirichlet condition u(1) = 0.6. The exact solution is still u(x) = 5x3 − 7.5x2 + 3x + 0.1. Using x = 0.01, the plots shown in Figure 13.2 compare the result using the method of ghost points and the result using the matrices in (13.28) and (13.29). For both methods the errors are within ±10−2 , with the direct method slightly better for this example.

496

Finite Difference Methods −3

3

x 10

2.5

2

Error

Figure 13.2. A plot of the errors from exact solution using the method of ghost points (solid line) and the direct method using matrices defined in (13.28) and (13.29) (dashed line).

1.5

1

0.5

0 0

0.2

0.4

0.6

0.8

1

x

13.2.2 Two-Dimensional Case To formulate the problem for the 2D case, we can collect the values of u in a K × N matrix U = (uij ) where the row indices refer to the x positions and the column indices refer to the y positions, that is, uij = u (i x, j x)

1 ≤ i ≤ K, 1 ≤ j ≤ N

and u0,j , uK+1,j , ui,0 and ui,N+1 refer to the boundary values. With this convention, we can put the terms corresponding to finite difference approximations of the partial derivatives in (13.36) in the following matrix forms: ∂u → D(1,x) U + B(1,x) ∂x

;

∂2u → D(2,x) U + B(2,x) ∂x2

∂u T T → UD(1,y) + B(1,y) ∂y

;

∂2u T T → UD(2,y) + B(2,y) ∂y2

(13.34)

Depending on boundary conditions and order of approximations, the matrices D(1,x) , D(2,x) , D(1,y) , and D(2,y) can take the forms of Di ’s given in (13.24) to (13.27) or (13.28) and (13.29), where the additional subscripts x and y simply indicate the sizes K × K and N × N, respectively. Conversely, the matrices B(1,x) , B(2,x) , B(1,y) , and B(2,y) are formed by augmenting the vectors bi ’s given in (13.24) to (13.27) or (13.28) and (13.29), where each column is evaluated at fixed values of the other variable. For instance, if the problem specifies Dirichlet conditions at both x = 0 and x = 1 while using the central difference formula, then ⎛ ⎞ −u0,1 −u0,N ⎜ ⎟ 0 0 ⎟       1 ⎜ ⎜ ⎟ . . .. . B(1,x) = = b(1,x) y1 · · · b(1,x) yN ⎜ ⎟ ··· . ⎟ 2 x ⎜ ⎝ ⎠ 0 0 uK+1,1 uK+1,N For the mixed derivative ∂ 2 u/(∂x∂y), the matrix formulation can be obtained by applying the partial derivatives in sequence, that is, ∂2u T → D(1,x) UD(1 + B(1,x,1,y) y) ∂x∂y

(13.35)

13.2 Time-Independent Equations

497

where T T B(1,x,1,y) = D(1,x) B(1,y) + B(1,x) D(1,y) + C(1,x,1,y)

and C(1,x,1,y) introduces data from extreme corner points of the domain. For instance, if the boundary conditions are all of Dirichlet types, while using the central difference formulas of first-order derivatives, we have ⎞ ⎛ u0,0 0 ··· 0 u0,N+1 ⎟ ⎜ 0 0 ⎟ ⎜ 1 ⎟ ⎜ . . .. . C(1,x,1,y) = ⎟ ⎜ 0 . ⎟ 4 x y ⎜ ⎠ ⎝ 0 0 uN+1,0 0 · · · 0 uN+1,N+1 With these matrix representations, we can now formulate the finite difference solution of a linear second-order linear differential equation given by μxx (x, y)

∂2u ∂2u ∂2u + μ (x, y) + μ (x, y) xy yy ∂x2 ∂x∂y ∂y2

+ βx (x, y)

∂u ∂u + βy (x, y) + ζ(x, y)u + η(x, y) = 0 ∂x ∂y

(13.36)

Similar to matrix U, we can evaluate the coefficients μxx , μxy , . . . at different spatial positions to construct the following K × N matrices: μxx , μxy , μyy , βx , βy , ζ, and η, where the row and column indices also refer to the x and y positions, respectively. Thus we have       T T T μxx ◦ D(2,x) U + B2,x + μxy ◦ D(1,x) UD(1,y) + B1,x,1,y + μyy ◦ UD(2,y) + B2,y     T T + βx ◦ D(1,x) U + B1,x + βy ◦ UD(1,y) + B1,y +ζ◦ U + η = 0 (13.37) where (◦) is the Hadammard product (or element-wise product). These can be further rearranged to formulate a linear equation by recalling the properties of matrix vectorization (cf. Table 1.4). Specifically, vec (A + B)

=

vec (BAC)

=

vec (A) + vec (B)  T  C ⊗ B vec (A)

vec (A ◦ B)

=

  vec(A) ◦ vec(B) = Adv vec(B)

(13.38)

where we use the superscript symbol “dv” to denote the diagonalized-vectorization operation, that is,    [  ]dv = diag vec  With (13.38), equation (13.37) can be reduced to the following matrix equation: R2D v = −f2D

(13.39)

498

Finite Difference Methods

where v = vec (U) and       R2D = μxx dv IN ⊗ D(2,x) + μxy dv D(1,y) ⊗ D(1,x) + μyy dv D(2,y) ⊗ IK     + βx dv IN ⊗ D(1,x) + βy dv D(1,y) ⊗ IK + ζdv f2D

=

         T μxx dv vec B(2,x) + μxy dv vec B(1,x,1,y) + μyy dv vec B(2,y)         T + βx dv vec B(1,x) + βy dv vec B(1,y) + vec η

Remarks: A MATLAB code that implements (13.39) based on Dirichlet boundary conditions and central difference approximation formulas is available on the book’s webpage as lin2D_centralfd_dirich.m. The function is invoked by the statement [u,x,y]=lin2D_centralfd_dirich(K,N) to obtain the solution u of size K × N at grid points x and y. The program will need to be be edited to customize its application with user-defined function coefficients μxx , . . . , and so forth and user-defined boundary conditions. EXAMPLE 13.5.

Consider the 2D Laplace equation ∂2u ∂2u + 2 =0 ∂x2 ∂y

subject to u(0, y) = g a (y), u(1, y) = g b(y), u(x, 0) = hc (x) and u(x, 1) = hd (x). In this case, μxx (x, y) = μyy (x, y) = 1 and the other coefficients are zero. Let us now use the central difference formulas ⎞ ⎞ ⎛ ⎛ −2 1 0 −2 1 0 ⎟ ⎟ ⎜ ⎜ . . ⎟ ⎟ 1 ⎜ 1 ⎜ 1 −2 . . 1 −2 . . ⎟ ⎟ ⎜ ⎜ and D(2,y) = D(2,x) = ⎟ ⎟ ⎜ ⎜ 2 2 . . . .

x ⎝

y ⎝ .. .. 1 ⎠ .. .. 1 ⎠ 0 1 −2 0 1 −2   ⎞ ⎛   ⎛ ⎞ hˆc hˆd (gˆa )1 (gˆa )N 1 1 ⎟ ⎜ ⎜ 0 ⎜ ⎟ ⎟ 0 0 0 ⎜ ⎜ ⎟ ⎟ ⎜ ⎜ ⎟ ⎟ .. .. .. .. B(2,x) = ⎜ and B = ⎜ ⎟ ⎟ (2,y) . ··· . . ··· . ⎜ ⎜ ⎟ ⎟ ⎜ ⎟ ⎝ 0 ⎠ 0 0 0 ⎝     ⎠ (gˆb)1 (gˆb)N hˆc hˆd K

K

with D(2,x) and D(2,y) having sizes K × K and N × N, respectively, and (gˆa )i = g a (i y)/ x2 , hˆc = hc (j x)/ y2 , . . . , and so forth. The matrices in (13.39) j

then become

R2D

⎛

AK

⎜ ⎜ sI ⎜ K =⎜ ⎜ ⎝ 0

sIK AK .. .

0 ..

.

..

.

sIK

sIK

AK

⎞ ⎟ ⎟ ⎟ ⎟ [=] (KN) × (KN) ⎟ ⎠

13.2 Time-Independent Equations

and

  (gˆa )1 + hˆc 1   ⎜ ⎜ ˆ h c ⎜ 2 ⎜ ⎜ .. = vec ⎜ ⎜  . ⎜ hˆc ⎜ ⎝ K−1   (gˆb)1 + hˆc ⎛

f2D

⎛

q ⎜ ⎜ r where AK = ⎜ ⎜ ⎝

(gˆa )N−1

  (gˆa )N + hˆd 1   hˆd

0

0

 hˆd

···

(gˆb)2

K

r ..

···

(gˆa )2

499

(gˆb)N−1

.. .

2

K−1 

(gˆb)N + hˆd

⎞



⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠

K

⎟ ⎟ ⎟, s = y−2 , r = x−2 and q = −2 (r + s). ⎟ .. .. . r ⎠ . 0 r q Note that R2D has a block tri-diagonal structure and AK has a tri-diagonal structure. These special structures allow for efficient methods such as the Thomas algorithm and the block Thomas algorithms discussed in Section 2.2.2 and Exercise E2.16, respectively, to be used.

EXAMPLE 13.6.

.

..

.

Consider the nonhomogeneous Helmholtz differential equation ∇ 2 u + 2u = g(x, y)

;

0 ≤ x, y ≤ 1

(13.40)

where

   πx     y 5 2 g(x, y) = 2x − 2x + + cos −2π cos (πy) + 5π2 − 8 sin (πy) 2 2 4

subject to u(0, y)

=

u(1, y)

=

1 − y sin (πy) 4 1 4

u(x, 0)

=

u(x, 1)

=

1 (2x − 1)2 4 1 (2x − 1)2 4

(13.41)

For this example, we know that the exact solution is given by    πx  1 2 u(x, y) = x − − y cos sin (πy) 2 2 Solving the matrix equations formulated in (13.39) with K = N = 99, that is, x = y = 0.01 (as implemented using the MATLAB function lin2D_centralfd_dirich.m), we have the plot of u(x, y) shown in Figure 13.3 together with a plot of the errors from the exact solution. Note that the errors are within ±10−4 .

The extension to three dimensions is straightforward but quite an increase in complexity. The formulation for the steady-state solutions of these types of problems, including examples, are given in Section M.2 as an appendix.

500

Finite Difference Methods

−5

x 10 4

0.5

2

Error

u(x,y) 0

0

−2 1

-0.5 1

1

1 0.5

y

0.5

0.5

0.5 0 0

0 0

y

x

x

Figure 13.3. The finite difference solution to (13.40) subject to conditions (13.41) is shown on the left plot, whereas the error from the exact solution in shown on the right plot.

13.2.3 Polar and Spherical Coordinates The finite difference methods are often applied to rectangular-shaped domains or domains that can be mapped into a rectangular domain. However, if the domain is chosen under polar coordinates (r, θ), the approach will require some modifications. First, the angular variable θ is limited to 0 ≤ θ ≤ 2π, where the values become periodic beyond this principal range. Second, the differential operators such as the Laplacian will yield formulas with rn in the denominators, that is, ∇ 2u =

∂ 2 u 1 ∂u 1 ∂2u + + 2 2 2 ∂r r ∂r r ∂θ

and complications arise at r = 0. To handle the issue of periodicity, we can incorporate the constraint u(θ = 0) = u(θ = 2π) to modify the matrices D1 and D2 in (13.24) and (13.25). Let n = 1, . . . , N + 1 with θ = 2π/(N + 1) and θ = (n − 1) θ. Then u(r, 0) = u(r, 2π)

→

uk,1 = uk,N+1

u(r, − θ) = u(r, 2π − θ) → uk,0 = uk,N (13.42)   Thus, with U = uk,n [=]K × N, uk,n = u (rk , θn ), the central difference formulas will yield  per T ∂u = U D1 ∂θ

and

 per T ∂2u = U D2 2 ∂θ

(13.43)

where ⎛ per D1

0

⎜ 1 ⎜ ⎜ −1 = 2 θ ⎜ ⎝ 1

−1

1 .. .

..

..

..

.

.

. −1

⎞

⎛

−2

⎟ ⎜ ⎟ ⎜ ⎟ and Dper = 1 ⎜ 1 2 ⎟ 2

θ ⎜ ⎝ 1 ⎠ 0 1

1 .. .

..

..

..

.

1

.

. 1

⎞

⎟ ⎟ ⎟ ⎟ 1 ⎠ −2 (13.44)

13.2 Time-Independent Equations

501

which are tri-diagonal matrices with additional terms at positions (1, N) and (N, 1). per per Note that both D1 and D2 are circulant matrices. There are many approaches to handling the complications at the origin. We discuss a simple approach that will stretch the domain by introducing fictitious points at r < 0 and a discretization that would bypass the origin yet make the fictitious points naturally disappear from the finite difference approximation equations.1 Assume that the radius has been normalized to be 0 ≤ r ≤ 1. Let k = 1, . . . , K with 2

r = and rk = (k − 21 ) r. Thus rK+1 = 1 and the fictitious points occur at 2K + 1 r0 = − r/2. With this choice and under the second-order central difference approximation of Laplacian at k = 1, the formula becomes  u2,n − 2u1,n + u0,n 1 u2,n − u0,n 1 u1,n+1 − 2u1,n + u1,n−1 ∇ 2 ur= r/2 ≈ + + 2

r

r/2 2 r

r2 /4

θ2 in which the terms involving u0,n do cancel out to yield  u2,n − 2u1,n 1 u2,n 1 u1,n+1 − 2u1,n + u1,n−1 ∇ 2 ur= r/2 ≈ + + 2

r ( r/2) 2 r r2 /4

θ2 Assuming Dirichlet conditions at r = 1 given by u(1, θ) = g(θ), we have  per T ∇ 2 u → D2 U + B2 + V D1 U + V B1 + WU D2 (13.45)   where U = uk,n , uk,n = u(rk , θn ), D2 and D1 are K × K matrices given by (13.24) per

and (13.25), respectively, D2 is given in (13.44), V and W are K × K diagonal matrices given by     1 1 1 1 V = diag ,..., W = diag 2 , . . . , 2 r1 rK r1 rK B2 =

1 1 H and B1 = H, with 2

r 2 r ⎛ ⎜ ⎜ H=⎜ ⎝

0 .. .

···

0 g(θ1 )

0 .. . 0 g(θN )

⎞ ⎟ ⎟ ⎟ ⎠

Thus for a Poisson equation ∇ 2 u = f (r, θ)  with F = f k,n , f k,n = f (rk , θn ), the finite difference equation reduces to a linear equation given by 

Au = b where u = vec (U) and

1

A

=

    per IN ⊗ (D2 + V D1 ) + D2 ⊗ W

b

=

vec (F ) − vec (B2 + V B1 )

(13.46)

This method is based on M. C. Lai, “A note on finite difference discretizations for Poisson equation on a disk,” Numerical Methods for Partial Difference Equations, vol. 17, 2001, pp. 199–203.

502

Finite Difference Methods

−3

x 10 2

1

Figure 13.4. The errors of the finite difference solution from the exact solution for the Laplace equation in polar coordinates given in Example 13.7.

Error 0

−1

−2 1

1 0

y

0

x

−1 −1

These results can be extended to handle the more general case of nonhomogeneous Helmholtz equation given by ∇ 2 u + h(r, θ)u = f (rθ) The finite difference solution for this case is included as an exercise in E13.7. The structure of A is a KN × KN, block-circulant matrix, which is a block tridiagonal matrix with additional blocks at the corners, that is, ⎛ ⎞ G H H ⎜ ⎟ ⎜ H G ... ⎟ ⎜ ⎟ A=⎜ ⎟ . . .. .. H ⎠ ⎝ H H G and there are efficient algorithms for solving problems with this special structure. (See, e.g., Section B.6.1, where the matrices can be split to take advantage of the block tri-diagonal inner structure.) Consider the Laplace equation in polar coordinates ∇ 2 u = 0 with boundary condition u(1, θ) = 1 + cos(3θ). The exact solution was found in Example 11.6 to be

EXAMPLE 13.7.

u = 1 + r3 cos(3θ) Based on (13.46), the errors of the finite difference solution from the exact solution are shown in Figure 13.4 while using K = 30 and N = 60 or r = 0.328 and θ = 0.1047. The errors are within ±2 × 10−3 . For the case of spherical coordinates, the Laplacian is given by ∇ 2u =

∂ 2 u 2 ∂u 1 ∂2u cos θ ∂u 1 ∂2u + + + + ∂r2 r ∂r r2 ∂θ2 r2 sin θ ∂θ r2 sin2 θ ∂φ2 per

per

(13.47)

The same matrices D1 and D2 defined in (13.44) are needed to satisfy the periodicity along the θ and φ variables. However, to handle the issue of the origin, there

13.2 Time-Independent Equations

503

is no need to change the discretization along r as was done for the case of polar or cylindrical coordinates. This means that we can set rk = k r, k = 1, . . . , K, where

r = 1/(K + 1). To see this, take the finite difference approximation of the terms in (13.47) that involve partial derivatives with respect to r at r = r1 = r  2  ∂ u 2 ∂u  u2,n,m − 2u1,n,m + u0,n,m u2,n,m − u0,n,m + ≈ +  2 2 ∂r r ∂r r=r1 = r

r

r2 =

2u2,n,m − 2u1,n,m

r2

Thus, for the case of azimuthal symmetry, that is, u = u(r, θ), we can use the regular discretizations k = 1, . . . , K, r = 1/(K + 1), rk = k r and n = 1, . . . , N,

θ = 2π/(N + 1), θ = (n − 1) θ to obtain a matrix equation similar to (13.45) that approximates the Laplacian operation in spherical coordinates, that is,  per T  per T + WU D1 Q (13.48) ∇ 2 u → D2 U + B2 + 2V D1 U + 2V B1 + WU D2 per

per

where the same definitions for D1 , D2 , B2 , B1 , D1 , D2 , V , and W used in (13.45) still apply, except for the main difference that rk = k r. The additional matrix Q is given by   cos θ1 cos θN Q = diag , ··· , sin θ1 sin θN As a final detail, note that because we have (cos θ1 / sin θ1 ) as the first term in Q, division by zero should be avoided by shifting the θ-grids by θ/2, that is, set θn = (n + 1/2) θ. Thus for a Poisson equation in spherical coordinates under azimuthal symmetry, ∇ 2 u = f (r, θ), and under a Dirichlet condition, u(1, θ) = g(θ), we can reduce the equation to a linear equation given by 



Au = b

(13.49)

where u = vec (U), F = f k,n , f k,n = f (rk , θn ), A

=

      per per IN ⊗ D2 + 2V D1 + D2 + QD1 ⊗W

b

=

vec (F ) − vec (B2 + 2V B1 )

Remarks: MATLAB codes that implement (13.46) and (13.49) for the 2D Poisson equation under polar and spherical (with azimuthal symmetry), respectively, are available on the book’s webpage as poisson2d_polar_dirich.m and poisson2d_polar_sphere.m, respectively. The function is invoked by the statement [U,r,th,x,y]=poisson2d_polar_dirich(K,N) (or [U,r,th,x,y]=poisson2d_sphere_dirich(K,N)) to obtain the solution U of size K × N at grid points x and y (or polar coordinates r=r and th=θ). The program will need to be be edited to customize its application with user-defined forcing function f (r, θ) and user-defined boundary conditions u(1, θ) = g(θ).

504

Finite Difference Methods

13.3 Time-Dependent Equations For partial differential equations that are dependent on time, the general form of finite-difference schemes is given by   u(q+1) = f u(q+1) , u(q) , u(q−1) , . . . , u(q−p )

(13.50)

where u(q) = u (q t) and contains the values of u(t, x) at the grid points. Equation (13.50) is called the time-marching scheme; that is, from an initial condition u(0) = u0 , (13.50) is evaluated iteratively as q is incremented. If the function f in (13.50) is not dependent on u(q+1) , the scheme is classified as explicit; otherwise, it is classified as an implicit. The integer p determines how many past values are used. For p > 0, we end up with a scheme that is known as a multilevel scheme.2 We limit our discussion only on linear time-marching schemes, that is, u(q+1) =

p 

(q)

αi u(q−i) + g (q)

(13.51)

i=−1 (q)

Thus, if α−1 = 0 for all q ≥ 0, the scheme is explicit.

13.3.1 The Semi-Discrete Approach By first replacing only the spatial partial derivatives with their corresponding finitedifference approximations, the original partial differential equation can be reduced to an initial value problem in the form of an ordinary differential equation given by M(t)

d u(t) =  F(t)u(t) +  B(t) dt

(13.52)

or assuming M(t) is nonsingular for t ≥ 0, d u(t) = F(t)u(t) + B(t) dt

(13.53)

When (13.53) is discretized with respect to time, this yields the time-marching equations of the form (13.51). This approach is known as the semi-discrete approach, also known as the method of lines. For instance, consider the following linear, time-varying, second-order equation in 2D space: αt

   ∂u ∂u ∂2u ∂2u + ϕts + μ ps + βs + ζu + η ∂t s=t,x,y ∂t∂s p,s=x,y ∂p∂s s=x,y ∂s

=

0

(13.54)

Using the procedures discussed in Section 13.2.2, the dependent variable u and the coefficients αt , ϕts , μ ps , βs , γ and η, with p, s = x, y, can be represented in matrix 2

More specifically, for p ≥ 0, we have a (p + 2)-level scheme in which two of the levels are for t = (q + 1) t and t = q t.

13.3 Time-Dependent Equations

505

form, for example, ⎛ U

=

u1,1 ⎜ .. ⎝ . uK,1

··· .. . ···

⎞ u1,N .. ⎟ ; . ⎠ uK,N

⎛

μxx

··· .. . ···

(μxx )1,1 ⎜ .. =⎝ . (μxx )K,1

⎞ (μxx )1,N ⎟ .. ⎠ . (μxx )K,N

; etc.

The terms with partial derivative in times then become ∂u ∂t

=

∂2u ∂t2

=

∂2u ∂t∂x

=

∂2u ∂t∂y

=

d U dt d2 U dt2     d d D(1,x) U + B(1,x) = D(1,x) U + dt dt     d d T T UD(1,y) + B(1,y) = + U D(1,y) dt dt

d B(1,x) dt d B(1,y) dt

(13.55)

d2 d (vec(U)) + M1 (vec(U)) + M0 (vec(U)) + N = 0 dt2 dt

(13.56)

After substitution to (13.54), we obtain M2 where =

M2 M1

=

M0

=

N

=

 dv ϕtt 

αt

dv

' (dv dv     + ϕtx IN ⊗ D(1,x) + ϕty D(1,y) ⊗ IK



R2D

' (dv  dv d d f2D + μtx B(1,x) + μty B(1,y) dt dt

and the terms R2D and f2D were given in (13.39). Next, let ⎛ ⎞ vec(U) ⎜ ⎟ ⎟ v=⎜ ⎝ d ⎠ (vecU) dt

(13.57)

For the special case of nonsingular M2 , we could rearrange (13.56) into semi-discrete form (13.53), that is, d v(t) = F(t)v(t) + B(t) dt

(13.58)

where  F=

I

0

−M−1 2 M1

−M−1 2 M0



 and

B=

0 −M−1 2 N



506

Finite Difference Methods

Another simple case occurs for diffusion problems. In this case, αt = −1 and ϕtt = ϕtx = ϕty = 0. Then M2 = 0, M1 = −1, M0 = R2D , and N = f2D . With v = vec(U), (13.56) reduces to d v = R2D v + f2D dt EXAMPLE 13.8.

(13.59)

Consider the linear reaction-diffusion differential equation given

by ∂u = μ∇ 2 u + ζu + η(t, x, y) ∂t

(13.60)

with boundary conditions u (t, 0, y) = v0 (t, y)

u (t, x, 0) = w0 (t, x)

u (t, 1, y) = v1 (t, y)

u (t, x, 1) = w1 (t, x)

(13.61)

and initial condition u (0, x, y) = ξ(x, y) where μ and ζ are constants. Let ⎞ ⎛ u1,1 . . . u1,N ⎜ .. ⎟ .. U = ⎝ ... . . ⎠ uK,1 . . . uK,N

⎛

η1,1 (t) ⎜ .. η(t) = ⎝ . ηK,1 (t)

(13.62)

... .. . ...

⎞ η1,N (t) ⎟ .. ⎠ . ηK,N (t)

with ukn = u(t, k x, n y) and ηk,n = η(t, k x, n y). Using the finite difference approximations used in (13.39), we obtain d v = R v + f(t) dt

(13.63)

where v

=

R

=

f

=

vec (U)   μ IN ⊗ D(2,x) + D(2,y) ⊗ IK + ζ INK     T vec η + μ vec B(2,x) + B(2,y)

which is an ordinary differential equation with an initial condition,   v(0) = vec ξ In this example, R is a constant matrix. However, in general, it can be timevarying.

Once the problem has been reduced to an initial-value problem, the methods discussed in Chapter 7, including Runge-Kutta and multistep methods can be employed. One concern, however, is that the size of the state vector can be enormous as the spatial-grid resolution increases to meet some prescribed accuracy. Moreover, matrix F in (13.53) will become increasingly ill-conditioned as the matrix size grows,

13.3 Time-Dependent Equations

507

0

−1000

Figure 13.5. The eigenvalues of R at different values of N + 1.

eigenvalues

−2000

−3000

−4000

−5000

−6000

−7000 5

10

15

N+1

as shown in Example 13.9 that follows. As discussed in Section 7.4.2, stiff differential equations favor implicit time-marching methods.

Consider the problem given in Example 13.8. For μ = 2, ζ = −3 and N = M (or x = 1/(N + 1) = y), matrix R becomes a block tri-diagonal matrix given by ⎞ ⎛ Ra Rb 0 ⎟ ⎜ ⎟ ⎜ Rb . . . . . . ⎟ ⎜ (13.64) R=⎜ ⎟ .. .. ⎝ . . Rb ⎠ 0 Rb Ra

EXAMPLE 13.9.

where, ⎛ ζ ⎜ ⎜ μ Ra = ⎜ ⎜ ⎝ 0

μ .. .

..

..

..

.

.

. μ

⎞

⎛ μ ⎟ ⎟ ⎜ ⎟ ; Rb = ⎝ ⎟ μ ⎠ 0 ζ 0

0 ..

. μ

⎞ 8 2 ⎟ ⎠ ; ζ=− 2 −3; μ=

x

x2

Figure 13.5 shows a plot of the eigenvalues of R at different numbers of grid points N + 1(= M + 1). Note also that all the eigenvalues are negative. Recall that the condition number ρ is given by the ratio ρ=

maxi (|λi |) min j (|λ j |)

where λi is an eigenvalue of R. Then ρ = 5.47 at N = 4 and grows to ρ = 150.1 at N = 19, which is often considered “mildly stiff”.

Instead of exploring various types of time-marching algorithms, we focus on the simplest one-step time-marching methods, known as the weighted-average Euler schemes.

20

508

Finite Difference Methods

13.3.2 Weighted-Average Euler Schemes Although it is true that the size of matrix F in (13.53) will grow as the number of grid points increases, the finite-difference approximations will still involve only the neighboring points. This means that the matrices are quite sparse. Several algorithms are available to take advantage of the sparsity properties. They usually involve iterative procedures for the evaluation of the time-marching schemes. Because of this, the most popular methods are the three types of one-step Euler methods: the forward-Euler (explicit), the backward Euler (implicit), and the midpoint-Euler (implicit). The last type is also known as the Crank-Nicholson method. From the initial-value problem (13.53) that was obtained from the semi-discrete approach: d v(t) = F(t)v(t) + B(t) dt subject to v(0) = v(0) , we have

1. Forward-Euler Schemes. v(q+1) − v(q)

t

=

v(q+1)

=

F(q) v(q) + B(q)   I + tF(q) v(q) + tB(q)

(13.65)

This scheme is an explicit type and is one of the easiest time-marching schemes to implement. Starting with the initial condition, v(0) , the values of v(q) are obtained iteratively. However, as with most explicit methods, stability is limited by the size of time steps used. Sometimes, the time steps required to maintain stability can be very small, resulting in a very slow time-marching process.3 Nonetheless, due to the ease of implementation, forward schemes are still used in several applications, with the caveat that stability may be a problem under certain parametric conditions. 2. Backward Euler Schemes. v(q+1) − v(q)

t

=

v(q+1)

=

F(q+1) v(q+1) + B(q+1) 

I − tF(q+1)

−1   v(q) + tB(q+1)

(13.66)

This is an implicit method and it requires inversion of (I − tF(q+1) ). Due to the sparsity in F, procedures such as LU factorizations, Thomas algorithms, or GMRES can be used to take advantage of the matrix structures (cf. Section 2.2). 3

In some cases, the required time steps may be too small such that round-off errors become very significant. In this case, the explicit scheme is impractical.

13.3 Time-Dependent Equations

509

For equations in two- and three-spatial dimensions, other modifications such as ADI schemes are also used (see the appendix given as Section M.4). Compared with the forward Euler schemes, the backward schemes are more stable. In some cases, the scheme will be unconditionally stable. This means that the scheme will be stable for any time step chosen. Of course, the solution accuracy still demand small time steps.4 3. Crank-Nicholson Schemes.      v(q+1) − v(q) 1  (q+1) (q+1) (q+1) (q) (q) (q) = F + F v +B v +B

t 2 9   8  

t (q+1) −1

t (q)

t  (q+1) (q+1) (q) (q) = I− I+ v F v + F B +B 2 2 2 (13.67) This is also an implicit method and again requires the inversion of (I − ( t/2)F(q+1) ). One advantage of Crank-Nicholson schemes over Euler backward schemes is an increase in the order of accuracy. It can be shown that the  accuracy of a Crank-Nicholson will be O t2 , xn , compared with the accuracy of a backward Euler scheme, which is O ( t, xn ). However, for discontinuous or non-smooth boundary conditions, the Crank-Nicholson method can introduce undesired oscillations unless the value t is small enough.5 For a simple exploration of the three methods applied to a one-dimensional diffusion equation, see Exercise E13.9. For equations in two- and three-spatial dimensions, other modifications such as ADI (alternate-dimension implicit) schemes are also used (see the appendix given as Section M.4). All three methods are special cases of an umbrella scheme called the Euler-θ method, which is also known as weighted-average Euler method. The weightedaverage Euler method is given by      v(q+1) − v(q) = θ F(q+1) v(q+1) + B(q+1) + 1 − θ F(q) v(q) + B(q)

t

(13.68)

From (13.68), we see that θ = 0, θ = 1/2, and θ = 1 yield the Euler-forward, CrankNicholson, and Euler-backward schemes, respectively.

EXAMPLE 13.10.

plane,

4 5

Consider the following time-dependent scalar field in the (x, y)  u(t, x, y) = e−2t ϕ(x, y) + 1 − e−2t ψ(x, y)

(13.69)

If the required solution is only the steady-state profiles, time accuracy may not be as important. In this case, large time steps can be used to speed up the convergence. Other additional techniques such as smoothing via averages can be used to reduce the amount of oscillations.

510

Finite Difference Methods

where, ϕ(x, y) =

8 1 + 20r(x, y)

r(x, y) = e2−2x−y

ψ(x, y) =

;

;

1 1 + 5s(x, y)

s(x, y) = e8[(x−0.8)

2

−y]

A linear reaction-diffusion type differential equation that has (13.69) as the solution is ∂u = 2∇ 2 u − 3u + g(t, x, y) (13.70) ∂t where g can be set by substituting (13.69) into (13.70). g(t, x, y)

=

q(x, y)

=

h(x, y)

=

f (ϕ + 2h(x, y) − ψ − 2q(x, y)) + (3ψ − 2h(x, y)) 32000 r2 3

−

800r

(1 + 20r) (1 + 20r)2 8  9 64 2 s2 3200 + 50 16x − 5 (1 + 5s)3 8   9 64 2 s − 400 + 5 16x − 5 (1 + 5s)2

Let the boundary conditions given by u (t, 0, y) = α0 (t, y)

=

u (t, 1, y) = α1 (t, y)

=

u (t, x, 0) = β0 (t, x)

=

u (t, x, 1) = β1 (t, x)

=

  e−2t ϕ(0, y) + 1 − e−2t ψ(0, y)   e−2t ϕ(1, y) + 1 − e−2t ψ(1, y)   e−2t ϕ(x, 0) + 1 − e−2t ψ(x, 0)   e−2t ϕ(x, 1) + 1 − e−2t ψ(x, 1)

(13.71)

and initial condition u (0, x, y) = ϕ(x, y)

(13.72)

This initial value-boundary condition problem satisfies the same situation given in Example 13.8. Thus the matrix formulation is given by d v = R v + f(t) dt Using the Crank-Nicholson method,      

t

t

t (q+1) (q+1) (q) (q) I− R v = I+ R v + f +f 2 2 2 8  −1  9  −1 

t

t

t

t (q+1) I− v I+ = I− R R v(q) + f(q+1) + f(q) R 2 2 2 2 With x = y = 0.05 (K = N = 19) and t = 0.001, the plots of the approximate solutions at different time slices are shown in Figure 13.6, together with the exact solution. The plots of the error distribution at different time slices are shown in Figure 13.7. The errors are within ±2 × 10−3 .

13.3 Time-Dependent Equations

t= 0.1

t= 0.2

1

u 0 1

y

0 0

x

1

t= 0.3

1

1

0.5

0.5

0 1

0.5

0 0

t= 0.4

0.5

1

0 1

0.5

0 0

t= 0.5 1

1

0.5

0.5

0.5

0.5

0 0

0.5

1

0 1

0.5

0 0

t= 0.7

0.5

1

0 1

0.5

0 0

t= 0.8 1

1

0.5

0.5

0.5

0.5

0 0

0.5

1

0 1

0.5

0 0

1

0.5

1

t= 0.9

1

0 1

0.5

t= 0.6

1

0 1

511

0.5

1

0 1

0.5

0 0

0.5

1

Figure 13.6. The finite difference solution to (13.70) at different slices of time, subject to conditions (13.71) and (13.72), using the Crank-Nicholson time-marching method. The approximations are shown as points, whereas the exact solutions, (13.69), at the corresponding t values are shown as a surface plots.

The backward Euler scheme is given by  −1   v(q+1) = I − tR v(q) + tf(q+1) For this example, the results of the backward Euler are similar to those of the Crank-Nicholson. This just means that even though one expectthe2Crank  would 2 2 Nicholson scheme to increase the accuracy from O t , x to O t , x2 , the accuracy of both schemes were still limited by the chosen x. The forward Euler scheme is given by   (q+1) = I + tR v(q) + tf(q) v However, time marching with the same step size t = 0.001, which was stable for Crank-Nicholson and backward schemes, was unstable for the forward scheme. As shown in Figure 13.8, the maximal absolute errors (near the edges) were greater than 4.0 at t = 0.014. The errors grow unbounded at further time steps. However, using a time step size t ≤ 0.00025 will produce a stable Euler-forward scheme.

512

Finite Difference Methods −3

−3

x 10

−3

x 10

t= 0.1

x 10

t= 0.2

t= 0.3

2 Error

2

0

0

−2 1

−2 1

−2 1

y

x

0 0

1

2

0.5

0 0

−3

0.5

1

0.5

0 0

−3

x 10

x 10

t= 0.5

t= 0.6

2

2

2

0

0

0

−2 1

−2 1

−2 1

0.5

0.5

0 0

1

0.5

0 0

−3

0.5

1

0.5

0 0

−3

x 10

t= 0.8

t= 0.9

2

2

0

0

0

0.5

0.5

0 0

1

1

x 10

2

−2 1

0.5

−3

x 10

t= 0.7

1

−3

x 10

t= 0.4

0.5

−2 1

0.5

0 0

0.5

1

−2 1

0.5

0 0

0.5

1

Figure 13.7. The error distribution between the finite difference approximation (using central difference formulas for spatial derivatives and Crank-Nicholson time-marching method) and the exact solutions, (13.69), at different t values.

13.4 Stability Analysis Three major issues that often need to be addressed by any numerical scheme are consistency, convergence, and stability. Although these three properties are closely related, they are strictly different.

t= 0.014

Error 5

Figure 13.8. The error distribution for finite difference approximation of u (using central difference formulas for spatial derivatives and Euler forward time-marching scheme with t = 0.001) from the exact solutions, (13.69), at t = 0.014.

0

−5 1

1 0.5

0.5

y 0

0

x

13.4 Stability Analysis

513

In this section, we limit our discussion to linear equations. Let the partial differential equation be Lv = f

(13.73)

where L is a linear partial differential operator and f is the nonhomogenous term. Based on (13.73), let the approximation schemes be described by L v = f

(13.74)

in which the subscript is attached with a refinement path that relates how the step size t and the other grid sizes such as x, y, and z are to be reduced.6 Definition 13.1. Let v p be the solution of (13.73) at point p , where p belongs to the discretized domain. If   (13.75) lim L v p − f = 0

→0

then scheme (13.74) is said to be consistent with (13.73).

Definition 13.2. Let v p and v p, be the solution of (13.73) and (13.74), respectively, at the same point p , where p belongs to the discretized domain. If       (13.76) lim  v p − v p,  = 0

→0

for all points p in the discretized space, then the scheme (13.74) is said to be convergent with (13.73). Definition 13.3. Suppose the homogeneous part of (13.74), that is, L u p, = 0, is rearranged into a two time-level formula given by (q+1)

v

(q)

= C(q) v

(13.77)

where v(q) is a vector containing u p, at all the points p = (q t, k x, n y, m z) in the discretized space. (0) (q) If for any initial condition v = v(0) , the vector v remains bounded as

→ 0, then (13.77) is said to be Lax-Richtmyer stable. Convergence is the most desirable property among the three because it aims for an accurate result for the unknown: u(t, x, y, z). Unfortunately, except for simpler equations, convergence of a proposed scheme can be very difficult to prove. However, consistency of a scheme is quite straightforward to show. Using Taylor series approximations, a proposed scheme will normally reveal the mismatch (L u p − f ) as a truncation error in terms of t, x, y and z. Thus, along the refinement path where → 0, the truncation errors will usually disappear if finite difference approximations of partial derivatives are used during the construction of the schemes. 6

For instance, one refinement path could be one that maintains the ratio t/( x y z) constant.

514

Finite Difference Methods

For Lax-Richtmyer stability, there exist several tests for necessary conditions, including spectral radius and von Neumann methods, as well as other sufficient conditions. In general, stability properties are still easier to prove than convergence properties. Fortunately, a crucial theorem called the Lax equivalence theorem states that for two-level linear consistent schemes, Lax-Richtmyer stability is both a necessary and sufficient condition for the convergence of a given scheme.7 Thus we describe next some of the tools of stability analysis. From these analyses, we can determine the range of time steps that maintains boundedness of the approximate solutions.

13.4.1 Eigenvalue Method for Stability Analysis Regardless of whether the semi-discrete approach was used or not, the finite difference time-marching schemes for a linear system can be represented by v(q+1) = C (q) v(q) + (q)

(13.78)

where (q) contains information from boundary conditions and nonhomogenous terms of the differential equation. For instance, for the Euler schemes discussed in Section 13.3.2, the ordinary differential equation resulting from discretization of spatial coordinates that yielded d v(t) = F(t)v(t) + G(t) dt can be put into the form given by (13.78), where the matrix C(q) is ⎧   I + tF(q) for forward Euler scheme ⎪ ⎪ ⎪ ⎪ ⎪ ⎨  −1 C(q) = I − tF(q+1) for backward Euler scheme ⎪ ⎪ ⎪ ⎪ ⎪ −1   ⎩  I − t F(q+1) I + t F(q) for Crank-Nicholson scheme 2 2 whereas for matrix (q) , ⎧

tG(q) ⎪ ⎪ ⎪ ⎪ ⎪ ⎨  −1

(q) = I − tF(q+1)

tG(q+1) ⎪ ⎪ ⎪ ⎪ ⎪ −1 t  (q+1)  ⎩  I − t F(q+1) G + G(q) 2 2

for forward Euler scheme for backward Euler scheme for Crank-Nicholson scheme

As long as matrix (q) is bounded, the stability of the finite difference scheme will only depend on matrix C(q) . For the case in which C is stationary, that is, not dependent on q, we have the following result. Let (q) be bounded for all q ≥ 0. Let λ = {λ1 , . . . , λm } be the set of distinct eigenvalues of C, with si copies of λi . A necessary condition for the scheme given in (13.78) to be numerically stable (i.e., |v(q) | < ∞ ) is that : ; max |λi | ≤ 1 (13.79)

THEOREM 13.1.

i=1,...,m

7

For a proof of the Lax equivalence theorem, see Morton and Myers (2005).

13.4 Stability Analysis PROOF.

515

Based on Section 3.6, a square matrix C can be factored to Jordan canonical

forms, C = T −1 JT where T is a nonsingular matrix, whereas J is a block diagonal matrix composed of m Jordan blocks, ⎛ ⎞ ⎞ ⎛ λi 1 0 0 J1 ⎜ ⎟ .. .. ⎜ ⎟ ⎟ ⎜ . . .. with J = J =⎝ ⎜ ⎟ [=] si × si ⎠ i . ⎝ λi 1 ⎠ Jm 0 λi Also, recalling formulas (3.36)-(2), the powers of J i are given by q

q

J i = P(q,si ) λi where

⎛ P(q,s)

βk,j

=

=

⎜ ⎜ ⎜ ⎝ ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩

1 0 .. .

β[q,q−1] λ−1 i 1 .. .

··· ··· .. .

β[q,q−s+1] λ−s+1 i β[q,q−s+2] λ−s+2 i .. .

0

0

···

1

k! (k − j )!j !

⎞ ⎟ ⎟ ⎟ [=] s × s ⎠

if j ≥ 0

0

otherwise

The homogeneous solution of (13.78) is then given by  −1 q (0) (q) vhomog = T JT v ⎛ q 0 P(q,s1 ) λ1 −1 ⎜ . .. = T ⎝ 0

⎞ ⎟ (0) ⎠T v q

P(q,sm ) λm

Thus max (|λi |) > 1 i

is sufficient to produce an instability for v(q) .

Condition (13.79) becomes both a necessary and sufficient condition for numerical stability when all the eigenvalues are distinct. The maximum absolute eigenvalue is also known as the spectral radius. Thus condition (13.79) is also known as the spectral radius condition. Note that numerical stability is only a necessary condition for Lax-Richtmyer stability. For Lax-Richtmyer stability, additional conditions are needed for the refinement paths; that is, numerical stability should be maintained as → 0. Nonetheless, for practical purposes, (13.79) usually satisfies the needs of most finite difference schemes. In fact, it can be shown that when C is a normal matrix, numerical stability becomes necessary and sufficient for Lax-Richtmyer stability.

516

Finite Difference Methods 2

max( | λ(C) | )

10

1

10

Δ t = 0.001

Figure 13.9. A plot of spectral radius of C in (13.80) at different values of t using forward Euler method.

0

10

Δ t = 0.00025

−1

10

−4

−3

10

10

Δt

−2

10

Because condition (13.79) depends on the eigenvalues, it is important to note that the calculation of eigenvalues usually contains inaccuracies. Thus one should still apply tolerance levels on the stability ranges for t, especially for large C matrices, and test the algorithms directly. For some banded matrices, such as tri-diagonal matrices, there exist explicit formulas for the eigenvalues.8 In Example 13.10, the forward Euler method was unstable for

t = 0.001 but was stable for t = 0.00025. The spectral radius for this scheme can be evaluated for

EXAMPLE 13.11.

C = I + t R

(13.80)

where R was defined in (13.64). With x = y = 0.05, K = N = 19, as used in Example 13.10, the spectral radius is plotted in Figure 13.9. From the figure, we can see that the spectral radius for t = 0.001 is greater than 1, whereas the spectral radius for t = 0.00025 is 1. Therefore, t = 0.001 should be unstable and t = 0.00025 should be stable because all the eigenvalues are distinct and C is a normal matrix. The curve in Figure 13.9 has a spectral radius greater than 1 when t > 3.20 × 10−4 . Conversely, by plotting the spectral radius of C for the backward Euler and the Crank-Nicholson methods, we see from Figure 13.10 that either of these methods will be stable for a wide range of t. In fact, from the plots, both the backward Euler and the Crank-Nicholson schemes appear to be unconditionally stable.

13.4.2 Von Neumann Method for Stability Analysis Although methods exists for the calculation of spectral radius of large sparse matrices, these methods are still computationally intensive. An alternative for stability analysis is to use a method known as the von Neumann method (also known as Fourier analysis method). Instead of taking the full matrix C into account, the von 8

See for instance, J. W. Thomas (1995).

13.4 Stability Analysis

517

1

Figure 13.10. A plot of spectral radius of C at different values of t for backward Euler and Crank-Nicholson methods.

max( |λ(C)| )

Crank−Nicholson

0.8

0.6

0.4

0.2

0 −4 10

Backward Euler −3

10

−2

10

Δt

−1

10

Neumann approach takes the point formulas, that is, the time-marching difference equation for u(q t, k x, n y, m z), and determines whether the difference scheme will amplify the magnitude of u at t + t at some points in the (x, y, z)-space. Specifically, the method proceeds as follows: 1. Set up the time-marching difference formula for u ((q + 1) t, k x, n y, m z) and consider only the homogeneous terms of the partial differential equation. 2. Set u to be   u q t, k x, n y, m z = ζq eiρx k x eiρy n y eiρzm z

(13.81)

√ where ζ, ρx , ρy and ρz are arbitrary real numbers. ( i = −1 ). 3. Substitute u given by (13.81) into the homogeneous time-marching scheme obtained from step 1. 4. Solve for ζ in terms of t, x, y and z. 5. Determine t such that |ζ| ≤ 1

(13.82)

for all possible values of ρx x, ρy y and ρz z     The value of ζ is known as the amplification factor. This method gives a necessary and sufficient condition for the stability of initial value problems or initialboundary value problems with periodic boundary conditions. However, it is only a necessary condition for problems with general nonperiodic boundary conditions. (See Exercise E13.11 for an example of a scheme in which the von Neumann condition is satisfied yet the scheme is found to be unstable for a process with nonperiodic boundary conditions.) Nonetheless, this method is one of the more straightforward approaches, and it is relatively easy to evaluate. It also provides a very good estimate of stable range for t, even in cases with nonperiodic boundary conditions.

0

10

518

Finite Difference Methods

Let us now use the von Neumann method to estimate the range of t that would yield a stable scheme for Example 13.10. With the forward Euler scheme, the time-marching scheme for (13.70)

EXAMPLE 13.12.

∂u = 2∇ 2 u − 3u + g(t, x, y) ∂t (q)

at point uk,n,m is  1  (q+1) (q) uk,n,m − uk,n,m

t

 2  (q) (q) (q) u − 2u + u k,n,m k−1,n,m

x2 k+1,n,m  2  (q) (q) (q) + uk,n+1,m − 2uk,n,m + uk,n−1,m 2

y   (q) − 3uk,n,m + g q t, k x, n y, m z

=

To consider only the homogeneous part, we remove g from the equation. Next, (q) we substitute (13.81) and then divide out uk,n,m from both sides to obtain 1 (ζ − 1)

t

=

4 (cos (ρx x) − 1)

x2    4  + 2 cos ρy y − 1 − 3

y

ζ

=

1 − (3 + 8μ) t

where μ=

    1

x 1

y 2 2 sin ρ + sin ρ ≥0 x y

x2 2

y2 2

Because the stability requirement should hold for the worst case situation, the values of t needed to keep |ζ| ≤ 1 can be determined by setting   3 + 8 max(μ) t ≤ 2 ρx ,ρy

or with max(μ) = (1/ x2 ) + (1/ y2 ),

t ≤

2 x2 y2 + 8 ( x2 + y2 )

3 x2 y2

By setting x = y = 0.05, we find that t ≤ 3.123 × 10−4 . This is comparable to the stability range found using the eigenvalue method (see Example 13.11), which is t ≤ 3.20 × 10−4 . For the backward Euler method, we have   1  (q+1) 2  (q+1) (q) (q+1) (q+1) uk,n,m − uk,n,m = u − 2u + u k+1,n,m k,n,m k−1,n,m

t

x2  2  (q+1) (q+1) (q+1) + u − 2u + u k,n,m k,n−1,m

y2 k,n+1,m   (q+1) − 3uk,n,m + g (q + 1) t, k x, n y, m z

13.5 Exercises

519

After removing g and substituting (13.81), the amplification |ζ| can be evaluated to be      1     ζ =  1 + (3 + 8μ) t  which means unconditional stability, because μ ≥ 0 and t > 0. Finally, for Crank-Nicholson schemes, the amplification formula can be found to be    1 − (3 + 8μ) ( t/2)      ζ =  1 + (3 + 8μ) ( t/2)  Because μ ≥ 0, we see that |ζ| ≤ 1 for t > 0. So once again, we have unconditional stability.

13.5 EXERCISES

E13.1. Obtain the coefficients for the finite difference formulas of the following and determine the order:   2   2   d2 u  d u 1  d u 1. = a + bu + e + cu + du k−1 k k+1  2 2 2 dx xk dx k−1 x dx2 k+1        d2 u  1 du d2 u 1 2. =a + buk + cuk+1 + d  2 2 dx xk

x dx k−1 x dx2 k+1 ∂2u , where ∂u/∂y ∂x∂y was approximated using the formula in item 7 of Table 13.1.

E13.2. Derive the approximation formula given in (13.21) for

E13.3. Find matrices D2 , D1 , b2 , and b1 (based on notations given in Section 13.2.1) for a fourth-order finite difference approximation that would handle the Neumann condition at x = 0 and Dirichlet condition at x = 1, that is, (du/dx)(0) = Neu0 and u(1) = Di1 . E13.4. Consider the following second-order differential equation: ∇ 2u + 2

∂2u ∂u ∂u + −2 − 9y = 0 ; ∂x∂y ∂x ∂y

0 ≤ x, y ≤ 1

subject to u(0, y) = −2y2 , u(1, y) = 1 + y − 2y2 , u(x, 0) = x2 and u(x, 1) = x2 + x − 2. Obtain the finite difference solution using the central difference approximations of all the partial derivatives, that is, using the formulation given in (13.39). Try using K = N = 100. Note: The exact solution for this problem is u = (x − y)(x + 2y). E13.5. Develop a finite difference solution of the steady-state linear 2D equation given in (13.34) using the matrix formulation of (13.39) that would handle the case where a Neumann boundary condition is fixed at y = 0, that is, ∂u (y = 0) = ψ0 (x). (Suggestion: One could modify the matrices for Di ’s and ∂y Bi ’s in the program lin2D_centralfd_dirich.m.) Test this program on the same problem given in  Exercise E13.4, except change the boundary ∂u  = x. The same exact solution applies, that condition at y = 0 to be ∂y  is, u = (x − y)(x + 2y).

y=0

520

Finite Difference Methods

E13.6. Use the method given in Section 13.2.3 to obtain the finite difference solution of the following Poisson equation in polar coordinates: ∇ 2 u = 4 − 21 cos (5θ)

0 ≤ r ≤ 1 , 0 ≤ θ ≤ 2π

subject to u(1, θ) = 2 + cos (5θ). Noting that the exact solution is given by u = 1 + r2 (cos (5θ) + 1), plot the solution and the error distribution for the case with K = 30 grid points along r and N = 200 grid points along θ. E13.7. Modify the formulation of matrix A in (13.46) to handle the generalization of the Poisson equation to the nonhomogeneous Helmholtz equation in polar coordinates given by ∇ 2 u + h(r, θ)u = f (r, θ) Create a program (or modify poisson2d_polar_dirich.m) to accommodate the change and test it on the case with h = 3, f = 18 − 15r3 sin(3θ) and boundary condition u(1, θ) = 6 − 5 sin(3θ). The exact solution for this case is given by u = 6 − 5r3 sin(3θ). Plot the error distribution for the case using K = 100 grid points along r and N = 100 grid points along θ. E13.8. Use the method given in Section 13.2.3 to obtain the finite difference solution of the following Poisson equation in spherical coordinates under azimuthal symmetry: 5 − 4 cos2 (θ) 0 ≤ r ≤ 1 , 0 ≤ θ ≤ 2π sin(θ) subject to u(1, θ) = 3 + sin (θ). Noting that the exact solution is given by u = 3 + r2 (sin (θ)), plot the solution and the error distribution for the case with K = 30 grid points along r and N = 100 grid points along θ. ∇ 2u =

E13.9. Consider the diffusion equation given by ∂u ∂2u = 2 ∂t ∂x subject to the conditions: u(x, 0) = 1, u(0, t) = 0, and u(1, t) = 0. 1. Recall the method of separation of variables and show that the analytical solution is given by  ∞   1 − cos (kπ) −(k2 π2 t) e u(x, t) = 2 sin (kπx) kπ k=1

2. Obtain the time-marching equation based on the central difference formula for ∂ 2 u/∂x2 ; that is, find F and B in (13.51) for this problem. 3. Using x = 0.01, try to obtain the finite difference solution from t = 0 to t = 0.1 using the three weighted-Euler methods, that is, the forward Euler, backward Euler, and Crank-Nicholson. First try t = 0.005 and then try t = 5 × 10−5 . Plot the time lapse solutions and error distribution of the solutions (if they are bounded). 4. Using the von Neumann stability method, show that the maximum time increment allowed for a stable marching of the forward Euler method for this problem will be t = x2 /2 (thus tmax = 5 × 10−5 will be stable.) 5. For the Crank-Nicholson method, the value of t need not be as small as the one needed for the stability of the forward Euler method to remove the oscillations. Try t = 5 × 10−4 and show that the oscillations are absent.

13.5 Exercises

521

E13.10. Consider the one-dimensional advection-diffusion equation ∂2u ∂u ∂u +α = 2 ∂x ∂x ∂t 1. Using the von Neumann stability method, obtain the amplification factor for the difference equation based on the central difference formulas for the spatial derivatives and forward Euler method for the time derivatives, in terms of α, t, and x. 2. Determine the maximum time increment t such that the forward-Euler method will be stable for α = −2 and x = 0.05. Verify your prediction for the initial and boundary conditions given by u(x, 0) = 0 and u(0, t) = u(1, t) = 1. E13.11. Consider the following scheme known as the Wendroff scheme: (q+1)

(q+1)

(1 + η) uk+1 + (1 − η) uk

(q)

(q)

= (1 − η) uk+1 + (1 + η) uk

(13.83)

where η = λ t/ x. 1. Show that, based on the von Neumann method, the amplification factor is given by |ζ| = 1 for all real η. 2. Use this scheme for λ = 0.5, t = 0.01 for a finite domain xk = k x, (q) (q)

x = 0.01 k = 0, . . . , 100, with Dirichlet conditions u0 = u100 = 0 and initial condition + 1 for 0.2 ≤ xk ≤ 0.4 (0) uk = 0 otherwise Note that this scheme will turn out to be unstable. Prove this instability using the eigenvalue analysis. 3. Change the previous case to have a periodic boundary condition and note that the system will become stable (although with growing oscillations). Show that the eigenvalue analysis for this case predicts stability. (This exercise shows that the von Neumann condition is only a necessary condition of stability for nonperiodic boundary conditions, but it becomes both necessary and sufficient for systems having only periodic boundary conditions. This is because the von Neumann method is based on Fourier analysis.) E13.12. Discuss how the matrices in (13.63) for Example 13.8 will have to be modified if the boundary conditions were changed to become ∂u (t, 0, y) = 0 ∂x ∂u (t, 1, y) = 0 ∂x after using the method of ghost points.

u (t, x, 0) = w0 (t, x) u (t, x, 1) = w1 (t, x)

E13.13. Write a general program that would solve the dynamic 2D Poisson equation in polar coordinates given by ∂u ∂ 2 u 1 ∂u 1 ∂2u + f (r, θ) = 2 + + 2 2 ∂t ∂r r ∂r r ∂θ subject to static initial and boundary conditions given by u(r, θ, 0) = U i (r, θ)

u(1, θ, t) = U R (θ)

522

Finite Difference Methods

where f (r, θ) is a forcing function. Test the program with the problem given in Example 11.8 and determine whether you get similar results to those shown in Figure 11.7. E13.14. Consider the one-dimensional time-dependent heat equation for a sphere that is symmetric around the origin given by  2  ∂T ∂ T 2 ∂T 2 =∇ T = + ∂t ∂r2 r ∂r with initial and boundary conditions given by T (r, 0) = 0, (∂T/∂r)r=0 = 0 and T (0, t) = 1. Based on the separation of variables method, the analytical solution is given by  ∞  2  (−1)n −α(nπ)2 t T (r, t) = 1 + e sin (nπr) (13.84) πr n n=0

1. Obtain the time-dependent temperature at the origin, T 0 (t), based on (13.84). 2. Use the central difference approximation and the Crank-Nicholson scheme to obtain a numerical solution of the temperature distribution using r = 0.05 and t = 0.001 for 0 ≤ r ≤ 1 and 0 ≤ t ≤ 0.5. Compare the values for T (0, tk ) to the function T (0, t) found in the previous problem and plot the errors. 3. Obtain the numerical solution using the Crank-Nicholson scheme under the same settings and problem as before, except modify the boundary condition at r = 1 to a Robin condition given by  ∂T  −β + T (1, t) = 1 ∂r  r=1

with β = 0.1. (Hint: You can use the method of ghost points to introduce this boundary condition while adding T (rK+1 , tq ) as unknowns.) E13.15. Use the von Neumann method to determine the stability region of the six basic finite difference schemes given in Table M.1. E13.16. Apply the same six finite difference schemes on the system given in Example M.3 but instead using an initial condition that contains a triangular pulse given by: ⎧ for 0.2 ≤ x ≤ 0.3 ⎨ 10x − 2 u(x, 0) = −10x + 4 for 0.3 < x ≤ 0.4 ⎩ 0 otherwise Observe whether significant oscillations will occur when using the CrankNicholson, leapfrog, or Lax-Wendroff schemes.

14

Method of Finite Elements

In this chapter, we discuss the finite element method for the solution of partial differential equations. It is an important solution approach when the shape of the domain (including possible holes inside the domains) cannot be conveniently transformed to a single rectangular domain. This includes domains whose boundaries cannot be formulated easily under existing coordinate systems. In contrast to finite difference methods which are based on replacing derivatives with discrete approximations, finite element (FE) methods approach the problem by piecewise interpolation methods. Thus the FE method first partitions the whole domain into several small pieces n , which are known as the finite elements represented by a set of nodes in the domain . The sizes and shapes of the finite elements do not have to be uniform, and often the sizes may need to be varied to balance accuracy with computational efficiency. Instead of tackling the differential equations directly, the problem is to first recast it as a set of integral equations known as the weak form of the partial differential equation. There are several ways in which this integral is formulated, including least squares, collocation, and weighted residual. We focus on a particular weighted residual method known as the Galerkin method. These integrals are then imposed on each of the finite elements. The finite elements that are attached to the boundaries of will have the additional requirements of satisfying the boundary conditions. As is shown later, these integrals can be reduced to matrix equations in which the unknowns are the nodal values. Because neighboring finite elements share the same nodes, the various “local” matrix equations have to be assembled to form the “global” matrix equation. Basically, the result is either a large linear matrix equation for steady-state problems or a large matrix iteration scheme for transient problems. There are several implementations of finite element method. In this chapter, we limit our discussion to the simplest approaches. First, we only tackle 2D linear second-order partial differential equations. The construction of the weak form for these problems is discussed in Section 14.1. We opted to tackle the 2D case because the 1D case can already be easily handled by finite-difference methods, whereas there are several 2D domains that are difficult to solve using only finite difference methods. With a good basic understanding of the 2D case, the extensions to three dimensions should be straightforward. Second, we limit our finite element meshes to be composed of triangles and apply only the simplest shape function (also known as interpolation function) based 523

524

Method of Finite Elements

on three vertices of the triangle. The various properties and integrals based on triangular elements are discussed in Sections 14.2 and 14.2.1, respectively. We include much later in Section 14.4 a brief description of mesh construction called the Delaunay triangulation. The use of triangles, although not the most accurate, simplifies the calculation of the integrals significantly compared with high-order alternatives. The inclusion of the boundary conditions depends on the types of conditions. The application of Neumann conditions and Robin conditions involves line integrals, which are discussed in Section 14.2.2. This will require another set of one-dimensional shape functions. After the weak form has been applied to the local elements to obtain the various matrix equations, the assembly of these local matrix equations to a global equation is discussed in Section 14.3. Once the assembly process has finished, only then are the Dirichlet conditions included. There are two approaches available for doing this: the matrix reduction approach and the overloading approach. Following most implementations of finite element methods, we focus on the overloading approach. Afterward, the various steps of the finite element method, based on triangular elements are summarized in Section 14.5. Having a basic description of the particular implementation of the finite element method using triangles for a linear second-order differential equation, we also include three extensions in this chapter. One is the improvement for convectiondominated cases known as the streamline upwind Petrov-Galerkin (SUPG) method, discussed briefly in Section N.2 as an appendix. Another extension is the treatment of axisymmetric cases discussed in Section 14.6, in which the techniques of the 2D finite element method can be used almost directly but with the inclusion of r as a factor inside the integrals. Finally, in Section 14.7, we discuss the use of the finite element method to handle unsteady state problems via the Crank-Nicholson method.

14.1 The Weak Form The general steps for the method of finite elements for the numerical solution of partial differential equations can be summarized as follows: 1. Reformulate the partial differential equation as an equivalent integral equation known as the weak form. 2. Decompose the domain into non-overlapping finite elements. 3. Evaluate the integral equations locally for each finite element, applying Neumann and Robin conditions when appropriate. 4. Assemble the local elements together and implement any Dirichlet boundary conditions to generate a global linear matrix equation. 5. Solve the matrix equation for the nodal values. We first limit our discussion to the solution of the following linear, second-order, elliptic partial differential equation:   ∂u ∂u ∂ 2 u ∂ 2 u F x, y, u, , , 2 , 2 ∂x ∂y ∂x ∂y

=

    ∇ · (M(x, y) · ∇u) + b(x, y) · ∇u + g(x, y)u + h(x, y) = 0 for (x, y) ∈

(14.1)

14.1 The Weak Form

525 u(Ω Ωn) u

Figure 14.1. Surface approximation using triangular finite elements.

y

Ωn

x

subject to the following boundary conditions:   M(x, y) · ∇u · n = q(x, y) on boundary ( )NBC (14.2)   M(x, y) · ∇u · n = μ(x, y)u + q(x, y) on boundary ( )RBC (14.3) or u

=

u(x, y)

on boundary ( )DBC

(14.4)

where is a region in the (x, y) plane, with the subscripts NBC, RBC, and DBC standing for Neumann, Robin, and Dirichlet boundary conditions, respectively, and n is the unit normal vector pointing outward of . The M tensor is assumed to be symmetric and positive definite. The functions u(x, y), q(x, y) and μ(x, y) are all assumed to be continuous functions along their respective boundary. As mentioned in the introduction, we limit our approach to the use of triangular finite elements and linear piecewise approximations. This means that the domain will be partitioned into triangular elements, each defined by the three vertices known as nodes. By assuming a linear-piecewise approximation of the surface in each element, a stitching of the neighboring elements will provide a continuous, albeit nonsmooth, surface solution to the partial differential equation. To illustrate, consider Figure 14.1. The finite element domain, n , is a triangular region in the (x, y)-plane defined by three vertices (or nodes), p1 , p2 , and p3 . The collection of u-values at each node, that is, u (p j ), will form the desired numerical solution to the partial differential equation. The linear approximation will then be a plane, denoted by u ( n ), as shown in Figure 14.1. Doing so, we see that the triangular planes will stitch continuously with their neighboring elements. Obviously, the approximation can be improved by increasing the order of approximation, that is, allowing for curved surfaces. When doing so, one needs to make sure that the surface elements will form a continuous surface when combined. In our case, we assume that increasing the resolution of the mesh will be sufficient for improving accuracy. In committing to use first-order approximations for each finite element, the first-order derivatives ∂u/∂x and ∂u/∂y will in general not be continuous across the shared edges of neighboring finite elements. This means that the second-order partial derivatives can no longer be evaluated around these neighborhoods. We need to reformulate the original partial differential equation to proceed with the first-order approximation. One such formulation is called the weak form. It transforms the original secondorder partial differential equation as given in (14.1) into an integral equation in which the integrands will involve only first-order partial derivatives. We begin by

526

Method of Finite Elements

including a weighing function on the right-hand side of (14.1). We can choose the variation of u, denoted by δu, as the weighing function,1  =

0

 =



δu F dA      δu ∇ · (M(x, y) · ∇u) dA + δu b(x, y) · ∇u dA  +





δu g(x, y)u + h(x, y) dA

(14.5)

Thus the solution is to find the values of u at the nodes that would make the sum of integrals in (14.5) as close to zero as possible, based on the chosen forms of the weights. An exact solution u(x, y) has the property that the values and derivatives of u will satisfy F = 0 as given in (14.1), while satisfying the boundary conditions. These are known as the strong solutions. By fixing the solutions to be composed of flat triangles, we are immediately setting the objective toward obtaining an approximate solution that is defined by the node values, and we do not expect F = 0, except in special circumstances. Instead, the factor inside the integrand that is weighted by δu can be taken instead as a “residual error” or “residual.” Thus the weak form given in (14.5) is also known as the weighted residual method. And the solution to (14.5) is known as a weak solution. The first integrand in (14.5) contains the second-order partial derivatives. As we had already mentioned, this causes a problem because the second derivatives of a surface composed of flat triangles will be zero inside each element, while becoming indeterminate at the edges of the elements. To alleviate this problem, we use two previous results: the divergence of a scalar product of a vector (cf. (4.55)) and the divergence theorem (cf. (5.5)), which we repeat below: 

∇ · ψv

=

F · n dS

=

ψ∇ · v + v · ∇ψ  ∇ · F dV

S

(14.6) (14.7)

V

Applying (14.6) to the integrand of the term in (14.5) that contains M,   δu ∇ · (M(x, y) · ∇u)

=

∇ · (δuM(x, y) · ∇u)   − ∇ (δu) · M(x, y) · ∇u

(14.8)

Next, using (14.7) (which is equivalently Green’s lemma for the 2D case) to the first term on the right-hand side of equation (14.8), 

1

 ∇ · (δu M(x, y) · ∇u) dA =

(δu M(x, y) · ∇u) · n ds

(14.9)

Bound( )

Instead of using the concept of “weight,” a similar approach is to treat δu as a “test function.”

14.2 Triangular Finite Elements

527

1

y

Figure 14.2. The shape function ψi (p).

pi

x

pk pj

where Bound ( ) is the boundary of and n is the outward unit normal at this boundary. Substituting (14.8) and (14.9) to (14.5), and then after some rearranging, we get   ⎞ ⎛ ∇ · M(x, y) · ∇u dA (δu)

  ⎟ ⎛ ⎜ ⎞ ⎟ ⎜ δu M(x, y) · ∇u · n ds ⎟ ⎜    ⎟ ⎜ Bound( ) ⎜ ⎟ − δu b(x, y) · ∇u dA ⎟ = ⎝ ⎜ ⎠ (14.10) ⎟ ⎜  ⎟ ⎜ + δu h(x, y) dA ⎠ ⎝    − δu g(x, y) u dA We refer to (14.10) as our weak-form working equation for the finite element method.

14.2 Triangular Finite Elements Consider the 2D finite element domain n , shown in Figure 14.1. This triangular domain is fixed by three distinct nodes, denoted by vectors p1 = (x1 , y1 )T , p2 = (x2 , y2 )T , and p3 = (x3 , y3 )T , with xi , yi ∈ , i = 1, 2, 3. Corresponding to each node, we can build three linear shape functions, ψ1 (x, y), ψ2 (x, y), and ψ3 (x, y), given by   det (p j − p) (pk − p)   ψi (p) = (14.11) det (p j − pi ) (pk − pi ) where i, j, k = {1, 2, 3}, i = j = k and p = (x, y)T is a point inside the triangle n formed by the three nodes pi p j and pk . Figure 14.2 shows the shape function ψi (p).

EXAMPLE 14.1.

Let the be the triangle whose vertices are given by       1 2 1.5 p2 = p3 = p1 = 1 1.5 2

then



=

2−x 1.5 − y  2−1 det 1.5 − 1

=

7 2 2 − x− y 3 3 3

det ψ1

 1.5 − x 2−y  1.5 − 1 2−1

ψ2

=

  1 − x 1.5 − x det 1−y 2−y   1−2 1.5 − 2 det 1 − 1.5 2 − 1.5

=

2 4 2 − + x− y 3 3 3

2 2 4 ψ3 = − − x + y 3 3 3

528

Method of Finite Elements

From this point on, we assume that the indexing of the points follow a cyclic sequence, that is, (i, j, k) = (1, 2, 3) , (2, 3, 1) , or (3, 1, 2) should trace a counterclockwise path. Let D/2 be the area of the triangle formed by pi , p j and pk , that is, (cf. (1.76) in Exercise E1.32), ⎛ ⎞ 1 1 1 ⎜ ⎟       ⎟ (14.12) D = det pi p j + det p j pk + det pk pi = det ⎜ ⎝ ⎠ pi p j pk Using the multilinearity property of determinants (cf. item 7 in Table 1.6), one can show that   D = det (p j − pi ) (pk − pi ) which is the denominator in (14.11). Thus with p constrained to be inside or on the triangle formed by pi , p j and pk , we have   2Area ppj pk   ψi = and 0 ≤ ψi (p) ≤ 1 2Area pi pj pk We now list some of the properties of the shape functions (with i, j, k ∈ {1, 2, 3} and i = j = k). These results can be obtained by direct application of (14.11). 1. ψi (p) has a maximum value of 1 at p = pi . ψ1 (p1 ) = 1 ;

ψ2 (p2 ) = 1

;

ψ3 (p3 ) = 1

(14.13)

2. ψi has a minimum value of 0 at p = λp j + (1 − λ)pk where 0 ≤ λ ≤ 1.   ψi λp j + (1 − λ)pk = 0

i = j = k

0≤λ≤1

;

(14.14)

3. The sum of all the three shape functions is 1. ψ1 (p) + ψ2 (p) + ψ3 (p) = 1

(14.15)

4. ψi is an affine function of x and y. Applying the multilinearity property of determinants to (14.11),     det p j pk p jy − p ky (p kx − p jx ) ψi (p) = x+ y+ D D D 5. The value of all shape functions is equal to 1/3 at p = 1 3

1 3

(p1 + p2 + p3 ).

1 (p1 + p2 + p3 ) 3   D . 6. The surface integral of ψi with respect to n is equal to 6  D ψi dA = 6

n

ψ1 (p∗ ) = ψ2 (p∗ ) = ψ3 (p∗ ) =

with p∗ =

(14.16)

(14.17)

(14.18)

Equation (14.18) is just the volume of the pyramid formed by the shape function ψi , as shown in Figure 14.2. The altitude of the pyramid is 1, whereas the area

14.2 Triangular Finite Elements

529 p

i

~

Figure 14.3. A linear (planar) approximation of f (x, y) with x and y in the region n .

p

i

n

of the triangular base has been determined earlier to be D/2. Thus because the volume of the pyramid is one-third the product of base area and the altitude, we obtain the value of D/6, where D is given by (14.12)

14.2.1 Surface Integrals of the Finite Elements Using the shape functions for each node in n , we now obtain formulas for evaluating the surface integrals in the working equation (14.10). For a continuous function f (x, y), a first-order approximation f˜ in n is a plane, as shown in Figure 14.3. This plane can be defined by the values of f (x, y) at the three vertices, that is, let f i = f (pi ), then f˜ = f 1 ψ1 + f 2 ψ2 + f 3 ψ3 

(14.19)

Based on (14.19), the surface integral in n is approximated as follows:     f (x, y)dA ≈ ψ1 dA + f 2 ψ2 dA + f 3 ψ3 dA f˜ (x, y)dA = f 1

n

n

=

D 2



f1 + f2 + f3 3



n

n

n

(14.20)

where we used (14.18) to evaluate the surface integrals of ψ1 , ψ2 , and ψ3 . Note that in (14.20), each f i represents one function evaluation. An alternative is to use the Gaussian quadrature approach and replace the three function evaluations with just one function evaluation. This means that instead of vertex values f 1 , f 2 , and f 3 , we need only one constant value, say f ∗ , evaluated at some point inside n . This point turns out to be p∗ = (1/3) (p1 + p2 + p3 ). Based on (14.17), f1 + f2 + f3 f ∗ = f˜ (p∗ ) = f 1 ψ1 (p∗ ) + f 2 ψ2 (p∗ ) + f 3 ψ3 (p∗ ) = 3 and the surface integral of f ∗ , held constant in n , yields     D ∗ D f1 + f2 + f3 ∗ ∗ f dA = f dA = f = 2 2 3

n

n

(14.21)

(14.22)

Thus the surface integrals given by (14.20) and by (14.22) are equal. The constant value, f ∗ , acts like a mean value of f˜ in n as shown in Figure 14.4. We will be replacing f (p) with several objects. In some cases, we will substitute f (p) with the elements of M(x, y) and b(x, y), the functions g(x, y) and h(x, y), as

530

Method of Finite Elements ~ ~

p* Figure 14.4. The volume in (a) is the same as the volume in (b). The volume in (a) is generated by the surface integral of f˜ = 3i=1 f i ψi . The volume in (b) is a prism of constant height and isgenerated by the surface integral of f˜ ∗ = 3i=1 f˜ (p∗ )ψi

n

p*

p* (a)

(b)

given in (14.10), even u and δu. Furthermore, we will be using f ∗ = f (p∗ ) – not f˜ (p∗ ), that is,  D f (x, y)dA ≈ f (p∗ ) (14.23) 2

n Strictly speaking, this is not exactly the same as (14.22) except when f (x, y) is flat in

n , for example, in the limit where the nodes of n approach each other. Nonetheless, because f˜ (x, y) is just a linear approximation of f (x, y) inside n to begin with, the integral approximation in (14.23) may not be worse than the integral approximation in (14.20). Letting f (x, y) be u in (14.19), we have u˜ = u1 ψ1 + u2 ψ2 + u3 ψ3 where ui = u (pi ). And at p∗ =

1 3

(14.24)

(p1 + p2 + p3 ), we can use (14.17) to evaluate u˜ (p∗ ),

u˜ (p∗ ) = u1 ψ1 (p∗ ) + u2 ψ2 (p∗ ) + u3 ψ3 (p∗ ) = ζT [u] where

⎞ u1 [u] = ⎝ u2 ⎠ u3 ⎛

⎛ ⎞ 1 1⎝ ζ= 1 ⎠ 3 1

and

(14.25)

(14.26)

Similarly, for δu, δu˜ ∗

δu˜ (p ) where

= =

δu1 ψ1 + δu2 ψ2 + δu3 ψ3 ∗

∗

(14.27) ∗

δu1 ψ1 (p ) + δu2 ψ2 (p ) + δu3 ψ3 (p ) = ζ [δu] T

(14.28)

⎞ δu1 [δu] = ⎝ δu2 ⎠ δu3 ⎛

A finite element approach in which the particular choice of the weight δu is formulated by (14.27) is known as the Galerkin method.2 It can be shown that for b = 0 2

Other weights can be chosen, which may indeed produce a more accurate solution, for example, the Petrov-Galerkin approach to be discussed later.

14.2 Triangular Finite Elements

531

and M that is symmetric, the Galerkin method yields an optimal solution of the weak form (14.5) for a given mesh. The gradients ∇u and ∇ (δu) will be approximated by ∇ u˜ and ∇ (δu) ˜ given below. Substituting (14.16) into (14.24) and rearranging, ⎛ ⎡ ⎞       p − p p − p p − p 2y 3y 3y 1y 1y 2y ⎟  ⎜ 1 ⎢ ⎢ x, y ⎜ ⎟ u˜ = ⎝ ⎣ ⎠ D (p 3x − p 2x ) (p 1x − p 3x ) (p 2x − p 1x )

+

=







det

' 

p2

x,



y

p3

,





det

p3

p1

T

=

1 D

det

p1

p2

⎤⎛

⎞ u1 ⎥⎝ u2 ⎠ ⎦ u3 (14.29)

−1 0

0 1





p1

p2

  1  det p2 p3 , D The gradient of u˜ then becomes

η

 

( T + η [u]

where, 

,



=

p3

⎛

0 ⎝ −1 1

 det



p3

p1

⎞ −1 1 ⎠ 0    det p1 p2

1 0 −1 ,

∇ u˜ = T [u]

(14.30)

∇ (δu) ˜ = T [δu]

(14.31)

Similarly, for δu, ˜

In summary, we have the following evaluations of the surface integrals in (14.10) for the n th finite element:

n

2   D [δu]T TT M(p∗ ) T [u] 2 n +   2 D [δu]T ζ bT(p∗ ) T [u] 2 n 2 +   D [δu]T g (p∗ ) ζζT [u] 2 n + 2   D [δu]T h(p∗ ) ζ 2 n

+

 ∇ (δu) · [M(x, y) · ∇u] dA

≈



n

δu [b(x, y) · ∇u] dA

≈

[δu g(x, y) u] dA

≈



n



n

δu h(x, y) dA

≈

(14.32)

14.2.2 Line Integrals Along the Boundary For the line integral in (14.10), we need to first identify whether two or more nodes of n belong to the boundary of the global domain . For cases (a) and (b) in Figure 14.5, that is, only one or none of the nodes of n belong to the boundary, the line integral will be set to zero. For case (c), the two collinear points that belong to boundary( ) will have values that contribute to the global line integral. In case

532

Method of Finite Elements

boundary

)

n

(a)

(b)

(c)

(d)

Figure 14.5. Possible attachments of n with boundary( ).

(d), all three vertices belong to boundary( ). However, for simplicity, we avoid this situation by constraining the mesh triangulation of to only allow cases (a), (b) or (c).3 Let pa and pb be two points of n that belong to boundary( ). As before, we can define shape functions for each point that would yield a linear approximation of the segment in the boundary. This time, the shape function for point pa and pb will be given by χa (p) χb(p), respectively, defined by     det pb p det pa p     (14.33) χa (p) = and χb(p) = det pb pa det pa pb where p ∈ pa pb with pa pb being the line segment connecting pa and pb given by   + 2 x = λpa + (1 − λ) pb , 0 ≤ λ ≤ 1 pa pb = y Figure 14.6 shows the shape function χb. Some of the properties of these shape functions are: χa (pa ) = χb(pb) = 1 ; χa (pb) = χb(pa ) = 0 ; χa (p) + χb(p) = 1     pa + pb pa + pb 1 χa = χb = (14.34) 2 2 2   pb − pa χa ds = χb ds = 2 pa pb pa pb Along points in pa pb , a continuous function f (x, y) can be approximated linearly by f (p) ≈ f (pa ) χa (p) + f (pb) χb(p) The line integral can then be approximated by  f (pa ) + f (pb) pb − pa f (x, y)ds ≈ 2 pa pb 3

(14.35)

(14.36)

For instance, when case (d) occurs, we can split the triangle into two triangles along the middle vertex such that the two smaller triangle elements satisfy case (c).

14.3 Assembly of Finite Elements

533

x,y

b

}

1

pb

y

Figure 14.6. The shape function χb.

pa n

papb x

where

" pb − pa =

(pb − pa )T (pb − pa )

Recalling the terms involving a line integral in (14.10), we can apply (14.36),     δua qa + δub qb  δu M(x, y) · ∇u · n ds ≈ pb − pa 2 Bound( n ) ⎧ ⎛ ⎞ ⎫ ⎪ ⎪ ⎪ ⎨ ⎬   ⎜ qa /2 ⎟ ⎪ T ⎜ ⎟ = L [δu] ea eb ⎝ ⎠ ⎪ (14.37) ⎪ ⎪ ⎪ qb/2 ⎩ ⎭ n

where δua = δu(pa ), δub = δu(pb),

  qa = (M(x, y) · ∇u) · n √ L = vT v

with

; p=pa

v=

  qb = (M(x, y) · ∇u) · n





p1

p2

and ei is the ith unit vector, that is, ⎛ ⎛ ⎞ ⎞ 1 0 e1 = ⎝ 0 ⎠ ; e2 = ⎝ 1 ⎠ 0 0

p3

eb − ea

p=pb



⎛

;

⎞ 0 e3 = ⎝ 0 ⎠ 1

In (14.37), we are assuming that the Neumann boundary conditions will supply the values of qa and qb. If this is not the case, the Dirichlet conditions will set the values of u at the boundaries – making the line integral calculations unnecessary. This result can be generalized easily for the Robin (i.e., mixed) conditions. The only change necessary is to replace qa and qb by      = μa u(pa ) + qa ; (M(x, y) · ∇u) · n = μbu(pb) + qa (M(x, y) · ∇u) · n p=pa

p=pb

where μa , μb, qa and qb should all be supplied at the boundary points with the Robin conditions.

14.3 Assembly of Finite Elements We now combine the results of the two previous sections. For the n th -finite element, the approximations of (14.32) and (14.37) can be grouped together based on the

534

Method of Finite Elements

weak-form (14.10) as follows: ⎛  ⎜ ⎜ ⎜ ⎜ ⎝

n

∇ (δu) · [M(x, y) · ∇u] dA

⎞

⎟ ⎟ − n δu [b(x, y) · ∇u] dA ⎟ ⎟ ⎠  − n [δu g(x, y) u] dA +   2 D ≈ (δu)T TT M(p∗ ) T − ζ bT(p∗ ) T − g (p∗ ) ζζT [u] 2 n 

= [δu]Tn Kn [u]n

(14.38)

and ⎛  ⎝

Bound( n )

+



n

(δuM(x, y) · ∇u) · n ds δu h(x, y) dA

⎞ ⎠

+ ≈ =

2  D T  ∗ h ζ + Q (δu) (p ) 2 n

[δu]Tn n

(14.39)

where Kn

=

n

=

+  2 D T T T T M(p∗ ) T − ζ b(p∗ ) T − g (p∗ ) ζζ 2 n +  2 D h(p∗ ) ζ + Q 2 n

(14.40) (14.41)

The constant matrices ζ and T were defined in (14.26) and (14.29), respectively, and ⎧    ⎪   qa L ⎪ ⎪ ea eb if pa , pb ∈ Bound( )NBC ⎨ D qb Q= (14.42) ⎪ ⎪ ⎪ otherwise ⎩ 0 The formulas for D and L were defined in (14.12) and (14.37), respectively. Note that the sizes of matrices Kn and n are 3 × 3 and 3 × 1, respectively. For the special case in which pa and pb both belong to the boundary points specified by Robin conditions, a term will have to be added to Kn as follows: +  2 D TT M(p∗ ) T − ζ bT(p∗ ) T − g (p∗ ) ζζT −  Kn = (14.43) 2 n  2 + D n = h(p∗ ) ζ + Q + Q(rbc) (14.44) 2 n where ⎛ ⎜ =⎜ ⎝





ea

eb

μa 0

0 μb



⎛

⎞⎞ T e ⎜ a ⎟⎟ L ⎜ ⎟⎟ ⎝ T ⎠⎠ D eb

14.3 Assembly of Finite Elements

and

Q(rbc) =

⎧ ⎪  ⎪ ⎪ ea ⎨

eb







q(rbc) a (rbc)

qb

⎪ ⎪ ⎪ ⎩ 0

L D

535

 if pa , pb ∈ Bound( )RBC otherwise

What we have done so and line integrals of the finite ? far is obtain the surface @ elements

n = . We now combine elements n in which ( i j = 0 for i = j ) and N n=1 the various integrals as follows: 

[♦] dA =

Nelements 



n

n=1

[♦] dA

and  [♦] ds =

Nelements 

Bound( )

 [♦] ds Bound( n )

n=1

where [♦] stands for the appropriate integrands. Using (14.38) and (14.39), applied to the weak-form working equation given in (14.10), we obtain Nelements 

[δu]Tn Kn [u]n =

Nelements 

n=1

[δu]Tn n

(14.45)

n=1

However, because [u]i and [u] j , with i = j , refer to different vectors, they cannot be added together directly. The same is true for [δu]i and [δu] j , i = j . Instead, we need to first represent (14.45) using u, the global vector of node values. To proceed at this point, assume that a mesh of triangular elements has been generated for domain . One particular method for mesh generations that would yield a triangulation is known as the Delaunay triangulation, which is discussed in Section 14.4. Regardless of the triangulation approach, we assume that the mesh is represented by two sets of data. One set of data is the collection of the node positions given by node matrix P,  P=

x1 y1

x2 y2

...

xNnodes yNnodes

 (14.46)

where Nnodes is the number of nodes in . The second set of data is represented by a matrix of indices (of the nodes) that make up each triangular finite element, given by index matrix I, ⎛ ⎜ ⎜ I=⎜ ⎝

I1,1 I2,1 .. .

I1,2 I2,2

I1,3 I2,3

INelements ,1

INelements ,2

INelements ,3

⎞ ⎟ ⎟ ⎟ ⎠

(14.47)

where Nelements are the number of elements in and Ii,j ∈ {1, 2, . . . , Nnodes } are the indices of the vertices of the ith triangular element. To illustrate,consider the mesh

536

Method of Finite Elements

y 2

2.0 1

1

2

3

1.0

Figure 14.7. Mesh triangulation involving 7 nodes and 6 elements. (The elements are identified by circled numbers.)

4

3

5 6

4

6

7

5

1.0

2.0

3.0

4.0

x

triangulation shown in Figure 14.7; then the corresponding matrices P and I are given by ⎛ ⎞ 1 3 2 ⎜ 2 3 4 ⎟ ⎜ ⎟   ⎜ 1 7 3 ⎟ 0.67 2.22 1.90 3.60 2.38 3.78 0.96 ⎟ P= and I = ⎜ ⎜ 3 7 5 ⎟ 1.63 2.13 1.25 1.43 0.31 0.45 0.29 ⎜ ⎟ ⎝ 3 5 4 ⎠ 4 5 6 Note that the sequence of the nodes in each row of I is such that it follows a counterclockwise path. For the n th element, we can use the rows of I to define a matrix operator En of size (Nnodes × 3) whose elements are given by + 1 if i = In,j En(i,j ) = (14.48) 0 otherwise When the transpose of En premultiplies a vector u of length Nnodes , the result is a (3 × 1) vector, [u]n , whose elements are extracted from positions indexed by {In,1 , In,2 , In,3 }.4 To illustrate, suppose Nnodes = 6 and In = {In,1 , In,2 , In,3 } = {6, 1, 3}, then ⎛ ⎞ u1 ⎞ ⎛ ⎞ ⎜ u2 ⎟ ⎛ ⎜ ⎟ u6 0 0 0 0 0 1 ⎜ ⎟ u 3 ⎟ ⎝ u1 ⎠ ETn u = ⎝ 1 0 0 0 0 0 ⎠ ⎜ ⎜ u4 ⎟ = ⎜ ⎟ u3 0 0 1 0 0 0 ⎝ u5 ⎠ u6 Returning to the main issue of summing the various terms in (14.45), we see that the (3 × 1) vector [u]n given by ⎛ ⎞ uIn,1 [u]n = ⎝ uIn,2 ⎠ (14.49) uIn,3 4

We can also use En to generate the matrix [p]n = (p1 , p2 , p3 )n from matrix P given in (14.46), that is,   x[In,1 ] x[In,2 ] x[In,3 ] [p1 , p2 , p3 ]n = P En = y[In,1 ] y[In,2 ] y[In,3 ] The local matrix [p]n can then be used to evaluate matrices Kn and n in (14.38) and (14.39).

14.3 Assembly of Finite Elements

can be extracted from the global vector ⎛ ⎜ ⎜ u=⎜ ⎝

537

⎞

u1 u2 .. .

⎟ ⎟ ⎟ ⎠

(14.50)

uNnodes using the matrix operator En , that is, [u]n = ETn u

(14.51)

[δu]n = ETn δu

(14.52)

The same is true for [δu]n ,

Substituting (14.51) and (14.52) into (14.45), we get Nelements 

[δu]

T

En Kn ETn u

n=1

N



[δu]

n=1



elements

T

=

Nelements 

En Kn ETn

[δu]T En n

N





elements

u

=

[δu]

T

n=1

En n

n=1 T

ˆ [δu] Ku

=

T

[δu] 

(14.53)

where  K

=

Nelements 

Kn(G)

n=1

 

=

Nelements 

n(G)

n=1

Kn(G)

=

En Kn ETn

n(G)

=

En n

Because the vector of variations δu is allowed to vary arbitrarily, we could remove it from both sides of (14.53). The assembled form is then given by,  = Ku 

(14.54)

One final detail still needs to be addressed. The Dirichlet conditions in (14.4) have not yet been included in (14.54). To address this issue, let the nodes that are attached to the Dirichlet conditions be indexed by vector D, D = (D1 , D2 , . . . , DNDBC )

(14.55)

where Di ∈ {1, 2, . . . , Nnodes } and NDBC is the number of nodes that are involved in the Dirichlet boundary conditions. We now describe two possible approaches. 1. Approach 1: Reduction of unknowns. In this approach, u is split into two vectors: uDirichlet , which contains the known u-values at the Dirichlet boundary, and

538

Method of Finite Elements

unonDirichlet , which contains the unknown u-values. Let vector Dnon be the vector of indices that remain after the removal of D, that is, ; : non (14.56) Dnon = D1non , . . . , D(N nodes −NDBC ) where Dinon ∈ \D

Dinon < Dnon j

and

if i < j

Then [uDirichlet ]i = u(Di )

i = 1, 2, . . . , NDBC

(14.57)

and [unonDirichlet ] j = u(Dnon j )

j = 1, 2, . . . , (Nnodes − NDBC )

(14.58)

 but with the rows D↓ be a matrix obtained from K With reference to (14.54), let K and columns that were indexed by D removed. Likewise, let  D↓ be the vector obtained from   but with elements that were indexed by D removed. T  Next, construct vector ϒ = ϒ1 , . . . , ϒ(Nnodes −NDBC ) using the known values of uDirichlet , ϒi =

N DBC 

(i,D ) [uDirichlet ] K

(14.59)

=1

Then we get the reduced set of simultaneous equations given by     D↓ [unonDirichlet ] =  K D↓ − ϒ

(14.60)

 Let K be the new matrix obtained from K  2. Approach 2: Overloading matrix K. whose elements are given by + ii if i = j = D ;  = 1, . . . , NDBC (1/) K (14.61) Kij =  Kij otherwise and  be the new vector obtained from   whose elements are given by + ii uD if i = D ;  = 1, . . . , NDBC (1/) K (14.62) i =  i otherwise The parameter  is chosen to be sufficiently small such that the Dth diagonal element dominates the other elements in matrix K. This will effectively set ui = ui if i ∈ D. For instance, one could fix a very small value such as  = 10−10 . The final equation then becomes Ku = 

(14.63)

Remarks: In the first approach, the number of unknowns is reduced by a count of NDBC . The tradeoff to these reductions is the additional bookkeeping of the new indices in unonDirichlet needed to map both unonDirichlet and uDirichlet back to u for postprocessing. The second approach of overloading the matrix avoids the additional re-mapping procedures, but lacks the advantage of a reduced number of unknowns.  for Moreover, the overloading approach preserves any symmetry present in K, example, when b = 0. Because the solution of a linear equation is easier with a

14.4 Mesh Generation circumcircle circumcircle

3 1

1

1

2 2

3

1 2 4 circumcircle

4 2

Figure 14.8. An example of two alternative triangulations.

symmetric matrix, the overloading approach has become the preferred route in several finite element solutions. For our discussion, we use the overloading approach due to its simpler form. Specifically, in (14.63), u refers to the same vector in (14.54).

14.4 Mesh Generation In this section, we discuss one approach of partitioning into non-overlapping triangular subdomains or finite elements, n , where n = 1, 2, . . . , Nelements . Specifically, we discuss an “unstructured” approach of constructing Delaunay triangles from a given set of nodes.5 A Delaunay triangulation is a mesh that has the property that the circumcircle of each finite element; that is, the circle formed by the vertices of each triangle will not enclose nor touch any other node of the domain. For example, Figure 14.8 shows one mesh as a Dalaunay triangulation and another that is not. One reason for the choice of the Delaunay criteria is that these meshes often minimize the occurrence of “spear-shaped” triangles. Also, several algorithms exist that use the circumcenters, that is, the centers of the circumcircles, to refine meshes to obtain guaranteed limits on both maximum and minimum angles of each triangles. However, we just focus on one procedure to achieve the Delaunay triangulation without the refinement methods. Basically, the procedure involves two major steps: 1. Generate nodes in the domain. (a) Boundary points are selected such that when the lines connect adjacent points, the resulting shapes are sufficiently close to those of the boundaries. (b) Nodes are then introduced inside the boundaries. Around the regions where solution surfaces are expected to be sharply varying (e.g., larger curvature), more points are needed. These include points near curved boundary regions as well. For regions that are flatter, fewer points may suffice. (Reducing points when the solution surfaces are flat can significantly improve computational efficiency). 5

The classical approach is the “structured” method of mapping quadrilateral subregions that make up the domain. The structured approach often yields more uniform patterns. However, these procedures sometimes demand several inputs and setup time from the users. Some versions also require the solution of associated partial differential equations.

539

540

Method of Finite Elements

(c) Internal nodes, that is, excluding the boundary points, may need to be moved around locally so that the nodes are mostly equidistant to each other. This step is called smoothing. We do not discuss these smoothing techniques, but some of the more popular methods include the Laplacian smoothing and the force-equilibrium method.6 Instead, for simplicity, we just generate the points as close to equal size in the desired domain. The result of the first part of the procedure is the matrix of node positions P, that is,  P=

x1 y1

x2 x2

...

xNnodes yNnodes

 (14.64)

2. Identify the Delaunay triangles. One of the simpler methods for finding the Delaunay triangles is to use a process known as node lifting. Let (xo , yo ) be a point near the center of domain . A paraboloid function defined by z(x, y) = (x − xo )2 + (y − yo )2

(14.65)

is used to generate a set of 3D points ⎛ ) *  P=  P1 ,  P2 , . . . ,  PNnodes

where

⎜  Pi = ⎝

xi yi

⎞ ⎟ ⎠

(14.66)

zi (xi , yi ) as shown in Figure 14.9. Next, a tight polygonal cover of the lifted nodes can be generated to form a convex hull of these nodes using triangular facets. One particular method, which we refer to as the simplified-QuickHull algorithm, can be used to obtain such a convex hull of the lifted points. Details of this algorithm are given in the appendix as Section N.1.7 When the nodes of each facet are projected down onto the (x, y) plane, the triangular mesh that is generated will satisfy the conditions of a Delaunay triangulation, thus identifying the nodes of a finite element from each projected facet. This can be seen in Figure 14.9, where a vertical circular cylinder enclosing a facet of the triangulated paraboloid generates a circumcircle of the finite element in the (x, y)-plane. Because this cylinder that cuts through the paraboloid will include only the three points of the triangular facet, the projected triangle in the (x, y)-plane will have a circumcircle that should not contain other nodes. 6 7

See, for example, P. O. Persson and G. Strang, A Simple Mesh Generator in MATLAB, SIAM Review, vol. 46, pp. 329–345, June 2004. The facets belonging to the top of the paraboloid will need to be removed. These facets are distinguished from the bottom facets using the property that their outward normal vectors are pointing upwards, that is, having a positive z-component.

14.5 Summary of Finite Element Method

541

z

Figure 14.9. Projection of the triangular facets of the paraboloid will yield a Delaunay triangulation in the (x, y) plane.

x

y

Using the nodes identified with each facet, the matrix I can now be given by ⎛ ⎞ I11 I12 I13 ⎜ ⎟ ⎜ ⎟ I21 I22 I23 (14.67) I=⎜ ⎟ ⎝ ⎠ ··· ··· ··· INelements ,1 INelements ,2 INelements ,3 As cited earlier, to have a consistent sign for the area calculations of D in (14.12), we need to order the indices of each finite element in such a way that it follows the same counterclockwise direction. Remarks: In MATLAB, a Delaunay triangular mesh can be obtained easily using the command ind=delaunay(px,py) or ind=delaunayn([px’,py’]), where px and py are row vectors for the x and y positions, respectively, of the boundary and internal nodes, whereas ind will yield the index matrix for I in the same structure given in (14.67).8 In addition, another function triplot(ind,px,py) can be used to generate a plot of all the mesh of triangles associated with index matrix ind and points defined by px and py. EXAMPLE 14.2. Consider the triangulation of a rectangular plate with a circular hole. The triangulation begins with the assignment of points at the boundaries (see Figure 14.10(a)). Then other points are added to fillin the region (see Figure 14.10(b)). Next, the convex hull algorithm was used to determine the triangles (see Figure 14.10(c)). Finally, the triangles inside the holes are removed (see Figure 14.10(d)). (A MATLAB file squared_annulus_ mesh_example.m is available on the book’s webpage. The results shown in Figure 14.10 are generated by [px,py,ind]= squared_annulus_ mesh_example(0.5,1.0,10), where the inner and outer radius are 0.5 and 1.0, respectively.)

14.5 Summary of Finite Element Method We now gather all the results obtained thus far. This summary is based on solving the linear partial differential equation given in (14.1), repeated here: [∇ · (M(x, y) · ∇u)] + [b(x, y) · ∇u] + g(x, y)u + h(x, y) = 0 8

We have found that the function delaunayn is more robust than delaunay.

542

Method of Finite Elements

(a) Discretize boundary

(b) Fill with points

(c) Form triangles

(d) Clear holes

Figure 14.10. Delaunay triangulation of a square with circular hole.

The methods are limited to linear approximations using triangular elements only. This summary also refers only to the use of the overloading technique for incorporating the Dirichlet boundary conditions. 1. Generate the mesh data. PGlobal :

I:

D: N: R:

matrix of node positions   Global Global pGlobal p · · · p = Nnodes 1 2

 where

pGlobal = i

xi yi



matrix of node indices of each element ⎞ ⎛ I1 ⎟ ⎜ I2 ⎟ ⎜ ⎟ ⎜ where In = (In1 , In2 , In3 ) = ⎜ .. ⎟ ⎠ ⎝ . INelements ; : set of indices of Dirichlet boundary nodes = D1 , D2 , . . . , DNDBC ; : set of indices of Neumann boundary nodes = N1 , N2 , . . . , NNNBC ; : set of indices of Robin boundary nodes = R1 , R2 , . . . , RNRBC

14.5 Summary of Finite Element Method

2. Calculate local matrices of each element. Let ζ = n th finite element, with n = 1, 2, . . . , Nelements ,  PnLocal

=

 Local  p1 n

 Local  p2 n

⎛ Dn

=

Tn

=

p∗n

=

Qn

QRBC n

n

=

=

=

⎞

1

⎜ ⎜ det ⎜ ⎝

1 Dn 



1 PnLocal −1 0

0 1

=

n

=



1

T

. For the

 Local  pi = pGlobal , i = 1, 2, 3 Ini n

⎟ ⎟ ⎟ ⎠

⎛ 0  Local  ⎝ −1 Pn 1

1 0 −1

⎞ −1 1 ⎠ 0

 PnLocal (ζ)

⎧ ⎪ ⎪  ⎪ ⎪ ⎪ ⎪ ⎨ ea ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎩

0

⎧ ⎪ ⎪  ⎪ ⎪ ⎪ ⎪ ⎨ ea ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎩

In,a and In,b ∈ Nn

if

otherwise ⎛ ⎞  qa (RBC)   ⎜ ⎟ Ln ⎟ eb ⎜ ⎝ (RBC) ⎠ Dn qb

0

⎧ ⎪ ⎪  ⎪ ⎪ ⎪ ⎪ ⎨ ea ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎩

⎛ ⎞  qa   ⎜ ⎟ Ln ⎟ eb ⎜ ⎝ ⎠ Dn qb

if

In,a and In,b ∈ Rn

otherwise ⎛  μa ⎜ eb ⎜ ⎝ 0

0

⎞⎛ ⎞ 0 ⎟ ⎜eTa ⎟  L  n ⎟⎜ ⎟ ⎠ ⎝ T ⎠ Dn μb eb

if

In,a and In,b ∈ Rn

otherwise

√ Ln = vT v , v = PnLocal (eb − ea ) ⎛ ⎛ ⎛ ⎞ ⎞ ⎞ 1 0 0 e1 = ⎝ 0 ⎠ , e2 = ⎝ 1 ⎠ , e3 = ⎝ 0 ⎠ 0 0 1   Dn TTn M(p∗n ) Tn − ζ bT(p∗n ) Tn − g (p∗n ) ζζT − n 2   Dn h(p∗n ) ζ + Qn + Q(RBC) n 2 where

Kn

1

1,



 Local  p3 n

where

1 1, 3

543

544

Method of Finite Elements

3. Form the global matrices (a) Assemble finite elements = K

Nelements 

En Kn ETn

 =

and

Nelements 

n=1

5 (En )ij =

where

En n

n=1

1

if i = Inj

0

otherwise

(Remarks: In MATLAB, once the local matrices Kn and n have been found, instead of using En , we could use the simpler command lines such as K(In,In) = K(In,In) + K_n ; Gamma(In) = Gamma(In) + Gamma_n ;  and   are updated iteratively from n = 1 where In is the n th row of I; thus K to Nelements .) (b) Include Dirichlet boundary conditions using the “overloading" approach

K



=

=

) * Kij

{i }

where

where

Kij =

i =

⎧ ii (1/)K ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎩  Kij

⎧ ii uD (1/)K ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎩  i

if i = j = D  = 1, . . . , NDBC 0 tol and α > tol w = Au;

h = U ∗ w;

p = w − Uh;

if α > tol U← ρ=



p  p

U

" r(i)2 + α2

;

c= 

ri ← ρ; ⎛ Q←⎝

I[i−1] 0

R← 

r = Qh; 

r(i) ; ρ

R 0

s=

γ ← Qi+1,1 end if End While Loop Solve for y using back-substitution: ⎞ Q1,1 ⎟ ⎜ Ry = β ⎝ ... ⎠ Qi,1 Evaluate the final solution: x = x(0) + Uy

α ρ

 r

⎞  0  ⎠ Q c s 0 −s c

⎛

α= p

0 1



Appendix B: Additional Details and Fortification for Chapter 2 local quadratic model

635

model trust region

x

k

Figure B.10. The trust region and the local quadratic model based on x(k) . The right figure shows the contour plot and the double-dogleg step.

B.13 Enhanced-Newton Using Double-Dogleg Method Like the line search approach, the double-dogleg method is used only when a full Newton update is not acceptable. The method will use a combination of two types of updates: 1. Gradient Descent Update   T = −J F x(k) δG k k

(B.98)

  −1 δN = −J F x(k) k k

(B.99)

2. Newton Update

Because the Newton update was based on a local model derived from a truncated Taylor’s series, we could limit the update step to be inside a sphere centered around x(k) known as the model-trust region approach, that is, with Mk > 0  k x ≤ Mk

(B.100)

Assuming the Newton step is the optimum local step, the local problem is that of minimizing a scalar function ϕk given by ϕk ( x)

= =

1 (Fk + J k x)T (Fk + J k x) 2     1 T 1 Fk Fk + FTk J k x + xT J kT J k x 2 2

(B.101)

Note that the minimum of ϕ ( x) occurs at x = −J k−1 Fk , the Newton step. The local model is shown in Figure B.10 as a concave surface attached to the point x(k) , whereas the trust region is the circle centered around x(k) . The double-dogleg procedure starts with the direction along the gradient, that is, a path determined by x = σδG k . This will trace a parabola along the surface of the quadratic model as σ increases from 0:  σ2      1 T   T T T T 2 = F + PG (σ) = ϕ σδG F F − σ F J J F J J F k k k k k k k k k k 2 k 2

636

xN[k]

Appendix B: Additional Details and Fortification for Chapter 2

x [k]

Newton

x [k+1] x [k] CP

x

Figure B.11. The double-dogleg method for obtaining the update xk+1 .

[k]

The minimum of this parabola occurs at σ∗ =

FTk J k J kT Fk  2 FTk J k J kT Fk

This yields the point known as the Cauchy point, xCP = x(k) + σ∗ δG k (k)

(k)

(B.102)

(k)

Note that if xCP is outside the trust region, xCP will need to be set as the intersection of the line along the gradient descent with the boundary of the trust region. In Figure B.10, the contour plot is shown with an arrow originating from x(k) but terminates at the Cauchy point. (k) The full Newton step will take x(k) to the point denoted by xNewton , which is the minimum point located at the center of the elliptical contours. The Cauchy point, full-Newton update point, and other relevant points, together with the important line segments, are blown up and shown in Figure B.11. (k) (k) One approach is to draw a line segment from xNewton to the Cauchy point xCP . Then the next update can be set as the intersection of this line segment with the boundary of the trust region. This approach is known as the Powell update, or the single-dogleg step. However, it has been found that convergence can be further improved by taking another point along the Newton step direction, which we denote (k) (k) by xN . The Dennis-Mei approach suggests that xN is evaluated as follows:   (k) −1 (k) (B.103) = x − ηJ F x(k) xN = x(k) + ηδN k k where

8 η = 0.2 + 0.8 σ

∗

  9 FTk J k J kT Fk FTk Fk

The double-dogleg update can then be obtained by finding the intersection (k) (k) between the boundary of the trust region and the line segment from xN to xCP as shown in Figure B.11, that is, (k)

(k)

x(k+1) = x(k) + (1 − ρ) xCP + ρxN

(B.104)

Appendix B: Additional Details and Fortification for Chapter 2

where ρ

=

a

=

b

=

c

=

−b +

√

b2 − ac a

(k)

(k) 2

(k)

 (k) T

xN − xCP 

xN − xCP (k) 2

xCP

(k)

xCP

− Mk2

and Mk is the radius of the trust region. In case the update does not produce satisfactory results, then the radius will need to be reduced using an approach similar to the line search method. To summarize, we have the following enhanced Newton with double-dogleg procedure: Algorithm of the Enhanced Newton’s Method with Double-Dogleg Search. 1. Initialize. Choose an initial guess: x(0)   2. Update. Repeat the following steps until either F x(k) ≤  or the number of iterations have been exceeded (a) Calculate J k . (If J k is singular, then stop the method and declare “Singular Jacobian.”) N (b) Calculate the δG k and δk . (cf. (B.98) and (B.99), respectively). (k)

(k)

(c) Evaluate points xCP and xN : (cf. (B.102) and (B.103)) (d) Evaluate the step change k x: (k)

(k)

k x = (1 − ρ) xCP + ρxN where ρ is obtained by (B.104). (e) Check if k x is acceptable. If   F x(k) + k x

2

  > F x(k)

2

+ 2αFTk J k k x

with α ∈ (0, 0.5) (typically α = 10−4 ), then update is unacceptable. Modify the trust region:   Mk ← max 0.1Mk , min ( 0.5Mk , λ  k x ) where λ = −

  F x(k) + k x

FTk J k k x   2 − F x(k)

2

and repeat from step 2c above. Otherwise, if acceptable, continue to next step. (f) Update x(k) : x(k+1) = x(k) + k x

 − 2FTk J k k x

637

638

Appendix B: Additional Details and Fortification for Chapter 2

f(x1,x2)

10

80

5

60

x2

40

x

0

x

20

10 5

0 10

0 -5

0

x2

-10 -10

x1

[k+2]

[k+3]

x

[k+1]

x

[k]

−5

−10 −10

−5

0

5

10

x1

Figure B.12. A surface plot of f (x1 , x2 ) of (B.105) is shown in the left figure. The right figure shows the contour plot and the performance of the enhanced Newton with double-dogleg method with the initial guess at (x1 , x2 ) = (4, −6).

Remarks: A MATLAB code nsolve.m is available on the book’s webpage that implements the enhanced Newton method, where the line-search method is implemented when the parameter “type” is set to 2. Also, another MATLAB code that uses the enhanced Newton method for minimization of a scalar function is available on the book’s webpage as NewtonMin.m, where the line-search method is implemented when the parameter type is set to 2.

EXAMPLE B.16.

Consider the multivariable function f (x1 , x2 ) = ζ21 + ζ22 + 2

where

(B.105)

  x1 x2 1 ζ1 (x1 , x2 ) = 5 tanh − + − 3 3 2 x2 ζ2 (x1 , x2 ) = 1 − 2 A surface plot of f (x1 , x2 ) is shown in Figure B.12. When the enhanced Newton with double-dogleg method was used to find the minimum of f (x1 , x2 ), we see in Figure B.12 that starting with (x1 , x2 )0 = (4, −6), it took only three iterations to settle at the minimum point of (x1 , x2 )∗ = (0.5, 2) which yields the value f = 2. Conversely, applying the line-search method, in this case with the same initial point, will converge to a different point (x1 , x2 ) = (−432, 2) with f = 27. A particular property of the function f (x1 , x2 ) in (B.105) is that the minimum is located in a narrow trough. When the line-search approach was used, starting at (x1 , x2 )0 = (4, −6), the first Newton step pointed away from (x1 , x2 )∗ = (0.5, 2). However, the double-dogleg method constrained the search to a local model-trust region while mixing the gradient search direction with the Newton direction. This allowed the double-dogleg method a better chance of locating the minima that is close to the initial guess.

Appendix B: Additional Details and Fortification for Chapter 2

B.14 Nonlinear Least Squares via Levenberg-Marquardt There are several cases in which the linear least-squares methods given in Section 2.5 are not applicable. In those cases, Newton’s methods can be used to find the leastsquares solution when the unknown parameters are in nonlinear form. We can formulate the nonlinear least squares as follows: min x

where r is the vector of residuals

1 r (x) 2

2

(B.106)

⎛

⎞ r1 (x1 , . . . , xn ) ⎜ ⎟ .. r (x) = ⎝ ⎠ . rm (x1 , . . . , xn )

with m ≥ n. For instance, suppose we wish to estimate parameters x = (x1 , . . . , xn )T of a nonlinear equation f (x, w) = 0 where w are measured variables, for example, from experiments. Assuming we have m sets of data given by w1 , . . . , wm , the residual functions are ri (x) = f (x, wi )

i = 1, . . . , m

One could apply Newton’s method directly to (B.106). However, doing so would involve the calculation of d2 r/dx2 ,  T    m d2 dr dr d 2 ri r = + ri 2 2 dx dx dx dx i=1

which is cumbersome when m is large. Another approach is to first linearize r around x0 , that is,   d  r(x) = r(x0 ) + r (x − x0 ) = r(x0 ) + J (x0 ) (x − x0 ) dx x=x0 where J is the Jacobian matrix given by ⎛ ∂r1 ⎜ ∂x1 · · · ⎜ ⎜ . .. . J (x0 ) = ⎜ . ⎜ . ⎜ ⎝ ∂rm ··· ∂x1

∂r1 ∂xn .. . ∂rm ∂xn

⎞   ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠  

x=x0

This transforms the nonlinear least-squares problem (B.106) back to a linear leastsquares problem (cf. Section 2.5), that is, min x−x0

1 r(x0 ) + J (x0 ) (x − x0 ) 2

whose solution is given by the normal equation, −1  T T x − x0 = − J (x J J (x r 0 ) (x0 ) 0 ) (x0 )

2

(B.107)

639

640

Appendix B: Additional Details and Fortification for Chapter 2

We obtain an iterative procedure by letting x(k) = x0 be the current estimate and letting x(k+1) = x be the next update. This approach is known as the Gauss-Newton method for nonlinear least-squares problem: −1 T  J k rk (B.108) x(k+1) = x(k) − J kT J k where

  J k = J x(k) ;

  rk = r x(k)

As it was with Newton methods, the convergence of the Gauss-Newton method may need to be enhanced either by the line-search method or by a model-trust region method. However, instead of the line search or the double-dogleg approach, we discuss another model-trust region method known as the Levenberg-Marquardt method. Recall, from Section B.13, that the model trust region is a sphere centered around the current value x(k) . The minimization problem can then be modified to be the constrained form of (B.107): min

k x

1 r k + J k k x 2

2

subject to

k x ≤ Mk

(B.109)

where k x = x(k+1) − x(k) is the update step and Mk is the radius of the trust region. From Figure B.10, we see that there is a unique point in the boundary of the trust region where the value of function in the convex surface is minimized. This observation can be formalized by the following lemma: Levenberg-Marquardt Update Form The solution to the minimization problem (B.109) is given by  −1 T

k x∗ = − J kT J k + μI J k rk

LEMMA B.2.

(B.110)

for some unique value μ ≥ 0. PROOF.

(See Section B.10.2.)

Lemma B.2 redirects the minimization problem of (B.109) to the identification of μ such that q (μ) = sμ − Mk = 0

(B.111)

where  −1 T sμ = − J kT J k + μI J k rk Note that we set μ = 0 if s0  < Mk . Also, the derivative of q (μ) is given by  −1 sTμ J kT J k + μI sμ dq  q (μ) = =− (B.112) dμ sμ Although the Newton method can be used to solve (B.111), the More´ method has been shown to have improved convergence. Details of the More´ algorithm are included in Section B.14.1.

Appendix B: Additional Details and Fortification for Chapter 2

To summarize, we have the Levenberg-Marquardt method: Algorithm of Levenberg-Marquardt Method for Nonlinear Least Squares. 1. Initialize. Choose an initial guess: x(0)   2. Update. Repeat the following steps until either r x(k) ≤  or the number of iterations have been exceeded (a) Calculate J k . (b) Calculate μ and sμ using the More´ algorithm. (c) Set k x = sμ and check if k x is acceptable. If   r x(k) + k x

2

  > r x(k)

2

+ 2αrTk J k k x

with α ∈ (0, 0.5) (typically α = 10−4 ), then update is unacceptable. Modify the trust region:   Mk ← max 0.1Mk , min ( 0.5Mk , λ  k x ) where λ = −

rTk J k k x     2 r x(k) + k x − r x(k)

2

− 2rTk J k k x



and repeat from step 2b above. Otherwise, if acceptable, continue to next step. (d) Update x(k) : x(k+1) = x(k) + k x Remarks: A MATLAB code for the Levenberg-Marquardt method (using the More` algorithm) for solving nonlinear least squares is available on the book’s webpage as levenmarq.m. EXAMPLE B.17.

Suppose we want to estimate the parameters a, b, c, d, and e of

the function:

  y = d exp ax2 + bx + c + e

to fit the data given in Table B.5. Applying the Levenberg-Marquardt method with the initial guess: (a, b, c, d, e) = (0, 0, 0, 0, 0), we obtain the estimates: (a, b, c, d, e) = (−0.0519, 0.9355, −1.1346, 0.0399, 2.0055). A plot of the model, together with data points, is shown in Figure B.13. We also show, in the right plot of same figure, the number of iterations used and the final value of the residual norm.

B.14.1 Appendix: More´ Method Algorithm of More´ Method to obtain μ: 1. Generate initial guess. μ(0)

⎧ ⎨ 0 M  = k−1 ⎩ μ|x=x(k−1) Mk

if k = 0 otherwise

641

642

Appendix B: Additional Details and Fortification for Chapter 2 Table B.5. Data for example B.17 x

y

x

y

x

y

0.1152 0.7604 1.2673 2.7419 3.4332 4.3088 4.8618 5.3687 6.4747

2.0224 2.0303 2.0408 2.1197 2.1803 2.2776 2.3645 2.4382 2.6303

6.5207 7.0276 7.5346 8.3180 9.4700 10.3917 11.2212 12.0507

2.6118 2.7355 2.7855 2.8539 2.8645 2.7934 2.6645 2.5487

12.6037 13.1106 13.7097 14.3548 15.0461 16.2442 17.5806 19.7465

2.4487 2.3750 2.2855 2.2013 2.1355 2.0671 2.0250 2.0039

2. Update. μ(j )

sμ(j −1) = μ(j −1) − Mk



  q  q μ=μ(j −1)

3. Clip μ between minimum and maximum values

μ

(j )

←

⎧ (j ) ⎪ ⎨ μ

if Lo j ≤ μ(j ) ≤ Hi j

!  ⎪ ⎩ max Lo j · Hi j , 10−3 Hi j

otherwise

where

Lo j

←

⎧ q(0) ⎪ ⎪ −  ⎪ ⎪ ⎪ q (0) ⎨    ⎪ ⎪  q ⎪ ⎪ μ −   , Lo j −1 ⎪ max ⎩ q μ=μ(j −1)

if j = 0

otherwise

2

3

10

2.8 0

y

10 2

2.6

|| r ||

2.4

−2

10

2.2

2 0

−4

5

10

x

15

20

10

0

10

20

Iteration

30

40

Figure B.13. The model together with the data given in Table B.5. On the right plot, we have the number of iterations performed and the corresponding norm of the residuals.

Appendix B: Additional Details and Fortification for Chapter 2

Hi j

←

⎧ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨

J kT rk Mk   (j −1) min Hi , μ ⎪ j −1 ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎩ Hi j −1

if j = 0   if q μ(j −1) < 0 otherwise

4. Repeat until:   sμ(j ) ∈ 0.9Mk , 1.1Mk

643

APPENDIX C

Additional Details and Fortification for Chapter 3

C.1 Proofs of Lemmas and Theorems of Chapter 3 C.1.1 Proof of Eigenvalue Properties r Property 1: Eigenvalues of triangular matrices are the diagonal elements. Let A be triangular then det (A − λI) =

N 

(aii − λ) = 0

i=1

Thus the roots are: a11 , . . . , aNN . For diagonal matrices, Aei = aii ei = λi ei Thus the eigenvectors of diagonal matrices are the columns of the identity matrix. r Property 2: Eigenvalues of block triangular matrices are the eigenvalues of the block diagonals. Let Aii be ith block diagonal of a block triangular matrix A, then det (A − λI) =

N 

(Aii − λI) = 0

i=1

or det (Aii − λI) = 0 r Property 3: Eigenvalues of αA is αλ. (αA) v = (αλ) v r Property 4: Eigenvalues of A and AT are the same.   Because det (B) = det BT ,   det (A − λI) = det (A − λI)T = det AT − λI = 0 Thus the characteristic equation for A and AT is the same, yielding the same eigenvalues. 644

Appendix C: Additional Details and Fortification for Chapter 3

r Property 5: Eigenvalues of Ak are λk . For k = 0, A0 = I and the eigenvalues are all 1’s. For k > 0, Ak v = Ak−1 (Av) = λAk−1 v = · · · = λk v For k = −1, assuming A is nonsingular, 1 v λ (Note: Property 7 implies that eigenvalues are nonzero for nonsingular matrices.) Then for k < −1, v = A−1 Av = λA−1 v

A−1 v =

→

Ak v = Ak−1 (Av) = λAk−1 v = · · · = λk v r Property 6: Eigenvalues are preserved by similarity transformations. Using the eigvenvalue equation for T −1 AT ,     = det T −1 det (A − λI) det (T ) det T −1 AT − λI =

det (A − λI)

Because the characteristic polynomials for both A and T −1 AT are the same, the eigenvalues will also be the same. If v is an eigenvector of A corresponding to λ then Av = λv →

−1 TBT  −1 v B T v

that is, T −1 v is a eigenvector of B. r Property 7:  λ = |A|. i Using the Schur triangularization, ⎛ λ1 ⎜ 0 ⎜ U ∗ AU = ⎜ . ⎝ .. 0

= =

λv  λ T −1 v

× λ2 .. .

··· ··· .. .

× × .. .

0

···

λN

⎞ ⎟ ⎟ ⎟ ⎠

where U is unitary and ‘×’ represent possible nonzero entries. After taking the determinant of both sides, N   ∗ U  |A| |U| = |A| = λi

r Property 8:  λ = tr (A). i Using Schur triangularization, ⎛ ⎜ ⎜ U ∗ AU = ⎜ ⎝

i=1

λ1 0 .. .

× λ2 .. .

··· ··· .. .

× × .. .

0

0

···

λN

⎞ ⎟ ⎟ ⎟ ⎠

After taking the trace of both sides. ∗

∗

tr (U AU) = tr (AUU ) = tr(A) =

N  i=1

λi

645

646

Appendix C: Additional Details and Fortification for Chapter 3

r Property 9: Eigenvalues of Hermitian matrices are real, and eigenvalues of skewHermitian matrices are pure imaginary. Let H be Hermitian, then (v∗ Hv)∗ = v∗ H∗ v = v∗ Hv which means v∗ Hv is real. Now let λ be an eigenvalue of H. Then Hv

=

λv

v∗ Hv

=

λv∗ v

Because v∗ v and v∗ Hv are real, λ has to be real.  be skew-Hermitian, then Similarly, let H  ∗ ∗   ∗ ∗ v =   ∗   v=−  v H v v∗ H v H  v is pure imaginary. Let   then which means  v∗ H λ be an eigenvalue of H, v H

=

 λ v

v  v∗ H

=

 λ v∗ v

 v is pure imaginary,  Because  v∗ λ has to be pure imaginary. v is real and  v∗ H r Property 10: Eigenvalues of positive definite Hermitian matrices are positive. Because H is positive definite, v∗ Hv > 0, where v is an eigenvector of H. Because v > 0, we must have λ > 0. However, v∗ Hv = λ |v|2 . r Property 11: Eigenvectors of Hermitian matrices are orthogonal. If H is Hermitian, H∗ H = H2 = HH∗ . Thus, according to Definition 3.5, H is a normal matrix. Then the orthogonality of the eigenvectors of H follows as a corollary to Theorem 3.1. r Property 12: Distinct eigenvalues yield linearly independent eigenvectors. Let λ1 , . . . , λM be a set of distinct eigenvalues of A[=]N × N, with M ≤ N, and let v1 , . . . , vM be the corresponding eigenvectors. Then Ak vi = λi Ak−1 vi = · · · = λki vi We want to find a linear combination of the eigenvector that would equal the zero vector, α1 v1 + · · · + αn vn = 0 After premultiplication by A, A2 , . . . , AM−1 , α1 λ1 v1 + · · · + αM λM vM

=

0

.. . v1 + · · · + αM λM−1 vM α1 λM−1 1

=

0

Combining these equations, ⎛



α1 v1

···

αM vM

⎜ ⎜ ⎜ ⎝

1 1 .. .

λ1 λ2 .. .

··· ··· .. .

λ1M−1 λ2M−1 .. .

1

λM

···

M−1 λM

⎞ ⎟ ⎟ ⎟ = 0[N×M] ⎠

Appendix C: Additional Details and Fortification for Chapter 3

The Vandermonde matrix is nonsingular if λ1 = · · · = λM (cf. Exercise E1.14). Thus α1 v1 = · · · = αn vM = 0 Because none of the eigenvectors are zero vectors, we must have α1 = · · · = αM = 0 Thus {v1 , . . . , vM } is a linearly independent set of eigenvectors.

C.1.2 Proof for Properties of Normal Matrices (Theorem 3.1) Applying Schur triangularization to A, ⎛ λ1 ⎜ ⎜ U ∗ AU = B = ⎜ ⎜ ⎝

b12 .. .

··· .. . .. .

0

b1,N .. . bN−1,N λN

⎞ ⎟ ⎟ ⎟ ⎟ ⎠

If A is normal, then B = U ∗ AU will also be normal, that is, B∗ B = (U ∗ AU)∗ (U ∗ AU) = U ∗ A∗ AU = U ∗ AA∗ U = (U ∗ AU) (U ∗ AU)∗ = BB∗ Because B is normal, we can equate the first diagonal element of B∗ B to the first diagonal element of BB∗ as follows: |λ1 | = |λ1 | + 2

2

N 

|b1k |2

k=2

This is possible only if b1k = 0, for k = 2, . . . , N. Having established this, we can now equate the second diagonal element of B∗ B to the second diagonal element of BB∗ as follows: |λ2 |2 = |λ2 |2 +

N 

|b2k |2

k=3

and conclude that b2k = 0, for k = 3, . . . , N. We can continue this logic until the (N − 1)th diagonal of B∗ B. At the end of this process, we will have shown that B is diagonal. We have just established that as long as A is normal, then U ∗ AU = , where  contains all the eigenvalues of A, including the case of repeated roots. Next, we can show that the columns of U are the eigenvectors of A,



AU •,1

···

U ∗ AU

=



AU 

=

U  λ1 U •,1

AU •,N

=

···

λN U •,N



or AU •,i = λi U •,i Because U is unitary, the eigenvectors of a normal matrix are orthonormal.

647

648

Appendix C: Additional Details and Fortification for Chapter 3

Now assume that a given matrix, say C[=]N × N, has eigenvectors ) * orthonormal ∗   , where {v1 , · · · , vN } λ corresponding to eigenvalues , that is, V , . . . , λ CV = 1 N       = diag λ1 , . . . , λN . ∗   

=

(V ∗ CV )∗ (V ∗ CV ) = V ∗ C∗ CV

 ∗ 

=

(V ∗ CV ) (V ∗ CV )∗ = V ∗ CC∗ V

∗  =  ∗ , we have Because  C∗ C = CC∗ This means that when all the eigenvectors are orthonormal, the matrix is guaranteed to be a normal matrix.

C.1.3 Proof That Under Rank Conditions, Matrix Is Diagonalizable (Theorem 3.2) Suppose λ1 is repeated k1 times. From the rank assumption, rank(λ1 I − A) = N − k1 means that solving (λ1 I − A) v = 0 for the eigenvectors contain k1 arbitrary constants. Thus there are k1 linearly independent eigenvectors that can be obtained for λ1 . Likewise, there are k2 linearly independent eigenvectors that can be obtained for λ2 , and so forth. Let the first set of k1 eigenvectors v1 , . . . , vk1 correspond to λ1 while the subsequent set of k2 eigenvectors vk1 +1 , . . . , vk1 +k2 correspond to eigenvalue λ2 , and so forth. Each eigenvector from the first set is linearly independent from the other set of eigenvectors. And the same can be said of the eigenvectors of the other sets. In the end, all the N eigenvectors obtained will form a linearly independent set.

C.1.4 Proof of Cayley Hamilton Theorem (Theorem 3.3) Using the Jordan canonical decomposition, A = TJT −1 , where T is the modal matrix, and J is a matrix in Jordan canonical form with M Jordan blocks, a0 I + a1 A + · · · + an AN = T (a0 I + a1 J + · · · + an J N )T −1 ⎛ ⎜ ⎜ =T⎜ ⎝

charpoly(J 1 ) 0 .. .

0 charpoly(J 2 ) .. .

··· ··· .. .

0 0 .. .

0

0

···

charpoly(J M )

⎞ ⎟ ⎟ −1 ⎟T ⎠

(C.1)

The elements of charpoly(J i ) are either 0, charpoly(λi ), or derivatives of charpoly(λi ), multiplied by finite scalars. Thus charpoly(J i ) are zero matrices, and the right-hand side of Equation (C.1) is a zero matrix.

Appendix C: Additional Details and Fortification for Chapter 3

C.2 QR Method for Eigenvalue Calculations For large systems, the determination of eigenvalues and eigenvectors can become susceptible to numerical errors, especially because the roots of polynomials are very sensitive to small perturbations in the polynomial coefficients. A more reliable method is available that uses the QR decomposition method. First, we have present the QR algorithm. Then we describe the power method, which is the basis for the QR method for finding eigenvalues. Finally, we apply the QR method.

C.2.1 QR Algorithm QR Decomposition Algorithm (using Householder operators): Given A[=]N × M.  = IN 1. Initialize. K = A, Q 2. Iterate. For j = 1, . . . , min(N, M) − 1 (a) Extract first column of K: u = K•,1 (b) Construct a Householder matrix: u1

←

u1 − u

H

=

I−

2 uu∗ u∗ u

(c) Update K: K ← HK :  [j,...,N],• ← HQ  [j,...,N],• (d) Update last (N − j ) rows of Q Q (e) Remove the first row and first column of K: K ← K1,1↓  if N > M: ←Q  [M+1,...,N],•↓ 3. Trim the last (N − M) rows of Q Q 4. Obtain Q and R: Q

=

∗ Q

R

=

 QA

C.2.2 Power Method

  Let square matrix A have a dominant eigenvalue, that is, |λ1 | > λ j  , j > 1. An iterative approach known as the power method can be used to find λ1 and its corresponding eigenvector v1 . Power Method Algorithm: Given matrix A[=]N × N and tolerance  > 0. 1. Initialize. Set w = 0 and select a random vector for v 2. Iterate. While v − w >  w

←

v

v

←

v

←

Aw v v

649

650

Appendix C: Additional Details and Fortification for Chapter 3 5

10

|| v

k+1

k

− v ||

0

10

−5

10

−10

10

0

5

10

15

Iteration ( k ) Figure C.1. Convergence of the eigenvector estimation using the power method.

1 ∗ v Av v∗ v A short proof for the validity of the power method is left as an exercise (cf. E3.24). The power method is simple but is limited to finding only the dominant eigenvalue and its eigenvector. Also, if the eigenvalue with the largest magnitude is close in magnitude to the second largest, then convergence is very slow. This means that convergence may even suffer for those with complex eigenvalues that happen to have the largest magnitude. In those cases, there are block versions of the power method.

3. Obtain eigenvalue:

EXAMPLE C.1.

λ=

Let A be given by

⎛

3 ⎝ A= 1 2

2 2 1

⎞ 1 3 ⎠ 3

the power method found the largest eigenvalue λ = 6 and its corresponding eigenvector v = (0.5774, 0.5774, 0.5774)T in a few iterations. The norm v(k+1) − v(k) is shown in Figure C.1.

C.2.3 QR Method for Finding Eigenvalues As discussed in Section 2.6, matrix A can be factored into a product, A = QR where Q is unitary and R is upper triangular. If we let A[ 1] be a similarity transformation of A based on Q, A[ 1] = Q∗ AQ = RQ

(C.2)

then A[ 1] simply has reversed the order of Q and R. Because the eigenvalues are preserved under similarity transformations (cf. Section 3.3), A and A[ 1] will have the same set of eigenvalues. One could repeat this process k times and obtain A[ k]

=

Q[ k] R[ k]

A[ k+1]

=

R[ k] Q[ k]

Appendix C: Additional Details and Fortification for Chapter 3

where the eigenvalues of A[ k] will be the same as those of A. Because R[ k] is upper triangular, one can show1 that A[ k] will converge to a matrix that can be partitioned as follows:   B C lim A[ k] = (C.3) 0 F k→∞ where F is either a 1 × 1 or a 2 × 2 submatrix. Because the last matrix is block triangular, the eigenvalues of A will be the union of the eigenvalues of B and the eigenvalues of F . If F [=]1 × 1, then F is a real eigenvalue of A; otherwise, two eigenvalues of A can be found using (3.21). The same process can now be applied on B. The process continues with QR iterations applied to increasingly smaller matrices until all the eigenvalues of A are found.

EXAMPLE C.2.

Consider the matrix ⎛

−1 ⎝ A= −1 1

1 0 1

⎞ 0 1 ⎠ 0

After approximately 33 iterations using the QR method described, we obtain ⎛ ⎞ −1.3333 1.1785 −0.4083 A[ 33] = ⎝ 0.9428 −0.6667 0.5774 ⎠ 0.0000 0.0000 1.0000 which means one eigenvalue can be found as λ1 = 1. For the remaining two eigenvalues, we can extract the upper left 2 × 2 submatrix and use (3.21) to obtain λ2 = 1 + i and λ3 = 1 − i.

Although the QR method will converge to the required eigenvalues, the convergence can also be slow sometimes, as shown in preceding example. Two enhancements significantly help in accelerating the convergence. The first enhancement is called the shifted QR method. The second enhancement is the Hessenberg formulation. Both of these enhancements combine to form the modified QR method, which will find the eigenvalues of A with reasonable accuracy. The details of the modified QR method are included in Section C.2.4.

C.2.4 Modified QR Method In this section, we discuss the two enhancements that will accelerate the convergence of the QR methods for evaluation of the eigenvalues. The first enhancement is to shift the matrix A k by a scaled identity matrix. Then second is to use Householder transformations to achieve a Hesseberg matrix, which is an upper triangular matrix, but with an additional subdiagonal next to the principal diagonal. 1

For a detailed proof, refer to G. H. Golub and C. Van Loan, Matrix Computations, 3rd Edition, 1996, John Hopkins University Press.

651

652

Appendix C: Additional Details and Fortification for Chapter 3

C.2.5 Shifted QR Method Instead of taking the QR decomposition of A k , one can first shift it as follows: & A k = A k − σ k I

(C.4)

where σ k is the (N, N)th element of A k . We now take the QR decomposition of & A k , & & k& R k A k = Q

(C.5)

& k + σ k I A k+1 = & R k Q

(C.6)

which we use to form A k+1 by

Even with the modifications given by (C.4), (C.5), and (C.6), A k+1 will still be a similarity transformation of A k starting with A 0 = A. To see this, A k+1

=

& k + σ k I & R k Q −1    & k & k + σ k I A k − σ k I Q Q

=

−1  & k & k A k Q Q

=

Note that these modifications introduce only 2N extra operations: the subtraction of σ k I from the diagonal of A k , and the addition of σ k I to the diagonal of & k . Nonetheless, the improvements in convergence toward attaining the form & R k Q given in (C.3) will be significant.

C.2.6 Hessenberg Forms The second enhancement to the QR method is the use of Householder operators to transform A into an upper Hessenberg form. A matrix is said to have the upper Hessenberg form if all elements below the first subdiagonal are zero, ⎞ ⎛ × × ··· × ⎜ .. ⎟ ⎜ × ... ... . ⎟ ⎟ ⎜ (C.7) H=⎜ ⎟ . . .. .. × ⎠ ⎝ 0 × × where “×” denotes arbitrary values. To obtain the upper Hessenberg form, we use the Householder operators U x−y given in (3.7), U x−y = I −

2 (x − y) (x − y)∗ (x − y)∗ (x − y)

which will transform x to y, as long as x = y . With the aim of introducing zeros, we will choose y to be ⎛ ⎞ x ⎜ 0 ⎟ ⎜ ⎟ y=⎜ . ⎟ ⎝ .. ⎠ 0

Appendix C: Additional Details and Fortification for Chapter 3

Two properties of Householder operators are noteworthy: they are unitary and Hermitian. The following algorithm will generate a Householder matrix H such that HAH will have an upper Hessenberg form. Also, because HAH is a similarity transformation of A, both A and HAH will have the same set of eigenvalues. Algorithm for Householder Transformations of A to Upper Hessenberg Form: Start with G ← A. For k = 1, . . . , (N − 2) wi = Gk+i,k ; i = 1, . . . , (N − k) 1. Extract vector w: 2. Evaluate H: ⎧ I[N] if w − y = 0 ⎪ ⎪ ⎨   H= I[k] 0 ⎪ ⎪ ⎩ otherwise 0 U w−y where,

3. Update G: End loop for k

y

=

U w−y

=



w

I−

0

···

2 w−y

2

0

T

(w − y) (w − y)∗

G←HGH

Because the Householder operators U v will be applied on matrices, we note the following improvements: Let β = 2/ (v∗ v), w1 = A∗ v, w2 = Av and γ = v∗ Av, 1. Instead of multiplication U v A, we use U v A = A − βvw∗1 . 2. Instead of multiplication AU v , we use AU v = A − βw2 v. 3. Instead of multiplication U v AU v , we use U v AU v = A − βvw∗1 + (γv − βw2 ) v∗ . The improvement comes from matrix-vector products and vector-vector products replacing the matrix-matrix multiplications. Remarks: In Matlab, the command H=hess(A) will obtain the Hessenberg matrix H from A. EXAMPLE C.3.

Let

⎛ ⎜ ⎜ A=⎜ ⎜ ⎝

3 0 0 0 0

−4 1 0 2 −2

0 0 −2 0 0

12 0 3 −5 6

12 2 3 −6 7

⎞ ⎟ ⎟ ⎟ ⎟ ⎠

Using the algorithm, the resulting Hessenberg form is ⎛ 3 −4 0 12 ⎜ 0 1 −1.4142 1 ⎜ 0 2.8284 1 −8.4853 G = HAH = ⎜ ⎜ ⎝ 0 0 0 1.6213 0 0 0 −0.6213

12 1 −8.4853 3.6213 −2.6213

One can check that the eigenvalues of G and A will be the same.

⎞ ⎟ ⎟ ⎟ ⎟ ⎠

653

654

Appendix C: Additional Details and Fortification for Chapter 3

Note that for this example, the resulting Hessenberg form is already in the desired block-triangular forms, even before applying the QR or shifted QR algorithms. In general, this will not be the case. Nonetheless, it does suggest that starting off with the upper Hessenberg forms will reduce the number of QR iterations needed for obtaining the eigenvalues of A.

C.2.7 Modified QR Method We can now combine both enhancements to the QR approach to determine the eigenvalues of A. Enhanced QR Algorithm for Evaluating Eigenvalues of A: r Initialize: 1. Set k = N. 2. Specify tolerance . 3. Obtain G, a matrix in upper Hessenberg form that is similar to A. r Reduce G: While k > 2   Case 1: (Gk,k−1  ≤ ). 1. Add Gk,k to the list of eigenvalues. 2. Update G by removing the last row and last column.     Case 2: (Gk,k−1  > ) and (Gk−1,k−2  ≤ ). 1. Add μ1 and μ2 to the list of eigenvalues, where √ √ −b + b2 − 4c −b − b2 − 4c μ1 = ; μ2 = 2 2 and

b

=

−(Gk−1,k−1 + Gk,k )

c

=

Gk−1,k−1 Gk,k − Gk,k−1 Gk−1,k

2. Update G by removing the last two rows and last two columns. Case 3: (|Gk,k−1 | > ) and (|Gk−1,k−2 | > ). Iterate until either Case 2 or Case 3 results: Let σ = Gk,k , 1. Find Q and R such that: QR = G − σI 2. Update G: G ← RQ + σI End While-loop r Termination: Case 1: G = [λ], then add λ to eigenvalue list. Case 2: G[=]2 × 2, then add μ1 and μ2 to the list of eigenvalues, where √ √ −b + b2 − 4c −b − b2 − 4c ; μ2 = μ1 = 2 2 and b = −(G11 + G22 )

;

c = G11 G22 − G21 G12

Appendix C: Additional Details and Fortification for Chapter 3 EXAMPLE C.4.

Let

⎛ ⎜ ⎜ A=⎜ ⎜ ⎝

1 2 1 2 0

2 1 0 2 0

0 0 −1 1 2

1 0 2 0 0

1 2 −1 3 1

655

⎞ ⎟ ⎟ ⎟ ⎟ ⎠

After applying Householder transformations H, we obtain G = HAH that has the upper Hessenberg form ⎛ ⎞ 1 −2 −0.1060 −1.3072 −0.5293 ⎜ −3 1.8889 −1.1542 2.3544 1.7642 ⎟ ⎜ ⎟ ⎟ 0 −1.0482 0.8190 0.6139 −0.4563 G=⎜ ⎜ ⎟ ⎝ 0 0 −1.2738 0.0036 2.9704 ⎠ 0

0

0

−0.8456

−1.7115

After ten iterations of the shifted-QR method, G is updated to be ⎛ ⎞ 4.2768 0.2485 −2.2646 2.2331 −5.7024 ⎜ 0 −1.8547 2.3670 −1.3323 0.2085 ⎟ ⎜ ⎟ ⎜ ⎟ 0 −1.5436 0.4876 1.0912 −0.0094 ⎜ ⎟ ⎝ 0 0 −0.2087 0.3759 −0.0265 ⎠ 0

0

0

0

−1.2856

and we could extract −1.2856 as one of the eigenvalues. Then the size of G is reduced by deleting the last row and column, that is, ⎛ ⎞ 4.2768 0.2485 −2.2646 2.2331 ⎜ 0 −1.8547 2.3670 −1.3323 ⎟ ⎟ G←⎜ ⎝ 0 −1.5436 0.4876 1.0912 ⎠ 0 0 −0.2087 0.3759 Note that along the process, even though G will be modified and shrunk, it will still have an upper Hessenberg form. The process is repeated until all the eigenvalues of A are obtained: −1.2856, 0.0716, −0.5314 ± 1.5023i, and 4.2768.

C.3 Calculations for the Jordan Decomposition In this section, we develop an algorithm for the construction of a modal matrix T that would obtain the Jordan decomposition of a square matrix A. The canonical basis, that is, the columns of T , is composed of vectors derived from eigenvector chains of different orders. Definition C.1. Given matrix A and eigenvalues λ, then an eigenvector chain with respect to λ, of order r is chain(A, λ, r) = (v1 , v2 , . . . , vr ) where (A − λI)r vr = 0 v j = (A − λI)v j +1

(A − λI)r−1 vr = 0 j = (r − 1), . . . , 1

(C.8)

656

Appendix C: Additional Details and Fortification for Chapter 3

Note: If the order of the chain is 1, then the chain is composed of only one eigenvector. Algorithm for Obtaining Chain (A,λ,r). 1. Obtain vector vr to begin the chain. (a) Construct matrix M,  (A − λI)r−1 M(λ, r) = (A − λI)r

−I 0



(b) Use Gauss-Jordan elimination to obtain Q, W, and q such that   I[q] 0 QMW = 0 0 (c) Construct vector h + 0 hj = a randomly generated number

j = 1, 2, . . . , q j = q + 1, . . . , 2n

(d) Obtain vr by extracting the first N elements of z = Wh. 2. Calculate the rest of the chain. v j = (A − λI)v j +1

j = (r − 1), . . . , 1

Note that as mentioned in Section B.2, the matrices Q and W can also be found based on the singular value decomposition. This means that with UV ∗ = M, we can replace W above by V of the singular value decomposition. Furthermore, the rationale for introducing randomly generated numbers in the preceding algorithm is to find a vector that spans the last (2n − q) columns of W without having to determine which vectors are independent. EXAMPLE C.5.

Let

⎛ ⎜ ⎜ A=⎜ ⎜ ⎝

3 0 1 0 0

0 3 0 0 0

0 0 3 0 0

0 1 0 2 0

1 1 0 0 3

⎞ ⎟ ⎟ ⎟ ⎟ ⎠

Using the algorithm, we can find the chain of order 3 for λ = 3, ⎛ 0 −1.2992 0.8892 ⎜ 0 −1.2992 1.1826   ⎜ ⎜ 0.8892 1.8175 chain(A, 3, 3) = v1 v2 v3 = ⎜ −1.2992 ⎝ 0 0 0 0 0 −1.2992 we can directly check that (A − λI)3 v3 = 0

(A − λI)2 v3 = 0

v2 = (A − λI) v3

v1 = (A − λI) v2

and

⎞ ⎟ ⎟ ⎟ ⎟ ⎠

Appendix C: Additional Details and Fortification for Chapter 3

657

To obtain the canonical basis, we still need to determine the required eigenvector chains. To do so, we need to calculate the orders of matrix degeneracy with respect to an eigenvalue λi , to be denoted by Ni,k , which is just the difference in ranks of succeeding orders, that is, Ni,k = rank(A − λi I)k−1 − rank(A − λi I)k

(C.9)

Using these orders of degeneracy, one can calculate the required orders for the eigenvector chains. The algorithm that follows describes in more detail the procedure for obtaining the canonical basis. Algorithm for Obtaining Canonical Basis. Given A[=]N × N. For each distinct λi : 1. Determine multiplicity mi . 2. Calculate order of required eigenvector chains. Let ( '  p i = arg min rank(A − λ1 I) p = (N − mi ) 1≤p ≤n

then obtain ordi = (γi,1 , . . . , γi,p i ), where + Ni,k pi γi,k = max(0, [Ni,k − j =k+1 γi,j ])

if k = p i if k < p i

where, Ni,k = rank(A − λi I)k−1 − rank(A − λi I)k 3. Obtain the required eigenvector chains. For each γi,k > 0, find γi,k sets of chain(A, λi , k) and add to the collection of canonical basis. One can show that the eigenvector chains found will be linearly independent. This means that T is nonsingular. The Jordan canonical form can then be obtained by evaluating T −1 AT = J . Although Jordan decomposition is not reliable for large systems, it remains very useful for generating theorems that are needed to handle both diagonalizable and non-diagonalizable matrices. For example, the proof of Cayley-Hamilton theorem uses Jordan block decompositions without necessarily having to evaluate the decompositions.

EXAMPLE C.6.

Consider the matrix A, ⎛ 3 0 ⎜ 0 3 ⎜ A=⎜ ⎜ 1 0 ⎝ 0 0 0 0

0 0 3 0 0

0 1 0 2 0

1 1 0 0 3

⎞ ⎟ ⎟ ⎟ ⎟ ⎠

658

Appendix C: Additional Details and Fortification for Chapter 3

then λi

mi

pi

Ni,k

ordi

2 3

1 4

1 3

[1] [2, 1, 1]

[1] [1, 0, 1]

Next, calculating the required chains: ⎛ ⎜ ⎜ chain(A, 2, 1) = ⎜ ⎜ ⎝

0 −0.707 0 0.707 0 ⎛

⎜ ⎜ chain(A, 3, 3) = ⎜ ⎜ ⎝

⎞

⎛

⎟ ⎟ ⎟ ⎟ ⎠

⎜ ⎜ chain(A, 3, 1) = ⎜ ⎜ ⎝

0 0 −1.2992 0 0

−1.2992 −1.2992 0.8892 0 0

0 −0.5843 −1.0107 0 0

0.8892 1.1826 1.8175 0 −1.2992

The modal matrix T can then be constructed as, ⎛ 0 0 0 −1.2992 ⎜ −0.7071 −0.5843 0 −1.2992 ⎜ 0 −1.0107 −1.2992 0.8892 T =⎜ ⎜ ⎝ 0.7071 0 0 0 0 0 0 0

⎞ ⎟ ⎟ ⎟ ⎟ ⎠

⎞ ⎟ ⎟ ⎟ ⎟ ⎠

0.8892 1.1826 1.8175 0 −1.2992

⎞ ⎟ ⎟ ⎟ ⎟ ⎠

The Jordan canonical form is ⎛ J =T

−1

⎜ ⎜ AT = ⎜ ⎜ ⎝

2 0 0 0 0

0 3 0 0 0

0 0 3 0 0

0 0 1 3 0

0 0 0 1 3

⎞ ⎟ ⎟ ⎟ ⎟ ⎠

C.4 Schur Triangularization and SVD C.4.1 Schur Triangularization Algorithm Given: A[=]N × N Initialization: Set GN = A. For m = N, N − 1, . . . , 2 Obtain λ, an eigenvalue of Gm , and its corresponding orthonormal eigenvector v. Using Gram-Schmidt algorithm (cf. Section 2.6), obtain an orthonormal set of (m − 1) vectors {w1 , . . . , wm−1 } that is also orthonormal to v.

Appendix C: Additional Details and Fortification for Chapter 3

Let Hm =



v

w1

···

wm−1

659

 ; then use

∗ Gm Hm Hm

 =

λ 0

bT Gm−1



to extract Gm−1 and construct U m as  Um =

I[N−m] 0

0 Hm



Calculate the product: U = U N U N−1 · · · U 2

C.4.2 SVD Algorithm 1. Apply the QR algorithm on A∗ A: (a) Initialize: D = A∗ A,V = I[M] ,   (b) Iterate: While vec D − diag(D) >  i. D = QR via QR algorithm ii. D ← RQ iii. V ← VQ (Note: Re-index D and V such √ that dk+1 > dk .) 2. Calculate singular values: σi = dii , i = 1, . . . , M. 3. Obtain U: Let r be the number of nonzero singular values. (a) Extract Vr as the first r columns of V . (b) Set r = diag (σ1 , . . . , σr ). (c) Calculate: U r = AVr −1 r . × − r) (d) Find Uq [=]N (M   such that Uq is orthogonal to U r . (e) Set U = U r Uq . + if i = j ≤ r σi 4. Form [=]N × M: ij = 0 otherwise

C.5 Sylvester’s Matrix Theorem Let A have all distinct eigenvalues. Let vk and w∗k be the right and left eigenvectors A, respectively, corresponding to the same kth eigenvalue λk , such that w∗k vk = 1. Then any well-defined matrix function f (A) is given by

THEOREM C.1.

f (A) =

N 

f (λk ) vk w∗k

(C.10)

k=1

The classic version of Sylvester’s matrix theorem gives equivalent formulations of (C.10), two of which are the following: f (A) =

N  k=1

 =k

f (λk ) 

(λ I − A)

=k (λ − λk )

(C.11)

660

Appendix C: Additional Details and Fortification for Chapter 3

and f (A) =

N  k=1

adj (λ I − A) f (λk )  =k (λ − λk )

(C.12)

The advantage of (C.11) is that it does not require the computation of eigenvectors. However, there are some disadvantages to both (C.11) and (C.12). One is that all the eigenvalues have to be distinct; otherwise, a problem arises in the denominator. To show that (C.10) can be derived from (3.35), we need to first show that the rows of V −1 are left eigenvectors of A. Let w∗k be the kth row of V −1 , then

V

⎛

AV

=

V

−1

=

V −1 ⎛ λ1 ⎜ ⎝ 0

A

⎞ w∗1 ⎜ .. ⎟ ⎝ . ⎠A w∗N

=

0 ..

. λN

⎞⎛

⎞ w∗1 ⎟ ⎜ .. ⎟ ⎠⎝ . ⎠ w∗N

or w∗k A = λk w∗N Thus w∗k is a left eigenvector of A. Using this partitioning of V −1 , (3.35) becomes ⎛

f (A)

=

= =





v1

···

vN

v1

···

vN

⎞⎛

⎞ w∗1 ⎜ ⎟ ⎜ .. ⎟ .. ⎠⎝ . ⎠ ⎝ . 0 f (λN ) w∗N ⎛ ∗ ⎞  N  w1   ⎟ T ⎜ f (λk ) ek ek ⎝ ... ⎠ k=1 w∗N

f (λ1 )

0

f (λ1 ) v1 w∗1 + . . . + f (λN ) vn w∗N

C.6 Danilevskii Method for Characteristic Polynomial There are several methods for the evaluation of eigenvalues. For smaller matrices, the characteristic polynomials are first determined, and then the roots are then calculated to be the eigenvalues. For larger cases, other methods can be used that bypass the determination of the characteristic polynomial. Nonetheless, there are situations in which the determination of characteristic polynomials becomes the primary goal, such as problems in which the Cayley-Hamilton theorems are used. One highly effective approach to finding the characteristic polynomial is the Danilevskii method. The main idea is to find sequences of elementary matrix operators (e.g., those used in Gaussian elimination) such that a nonsingular matrix S can be used to transform a square matrix A into a lower block triangular matrix in which the block diagonal matrices are in the form of companion matrices.

Appendix C: Additional Details and Fortification for Chapter 3

661

Definition C.2. A square matrix C is said to be a companion matrix to a monic polynomial p (s) = sn + αn−1 sn−1 + . . . + α1 s + α0 if it has the form

⎛ ⎜ ⎜ ⎜ C=⎜ ⎜ ⎝

⎞

0 0 .. .

1 0 .. .

0 1 .. .

··· ··· .. .

0 0 .. .

0 −α0

0 −α1

0 −α2

··· ···

1 −αn−1

⎟ ⎟ ⎟ ⎟ ⎟ ⎠

(C.13)

It is left as an exercise (cf. E3.8) to show that the characteristic equation of C defined in (C.13) will be p (s) = sn + αn−1 sn−1 + . . . + α1 s + α0 = 0

(C.14)

Furthermore, each distinct eigenvalue λ of C has a corresponding vector given by ⎞ ⎛ 1 ⎜ λ ⎟ ⎟ ⎜ (C.15) v=⎜ . ⎟ ⎝ .. ⎠ λn−1 Thus with a similarity transformation of A based on a similarity transformation by S ⎛ ⎞ C1 ··· 0 ⎜ Q21 ⎟ C2 ⎜ ⎟ S−1 AS = ⎜ . (C.16) ⎟ .. .. ⎝ .. ⎠ . . Qr1 Qr2 · · · Cr where Ci are n i × n i companion matrices to polynomials p i (s) = sni + αni −1 sni −1 + · · · + α1 s + α0 [i]

[i]

[i]

the characteristic polynomial of A is then given by charpoly(A) =

r 

p i (s)

(C.17)

i=1

To find S, we have the following recursive algorithm:

Danilevski Algorithm: Let A[=]N × N; then Danilevski(A) should yield matrix S such that (C.16) is satisfied. Initialize k = 0 and S = IN While k < N, k←k+1 If N = 1, S=1

662

Appendix C: Additional Details and Fortification for Chapter 3

else      Let j max = arg max j ∈{i+1,...,N}  aij  and q = ai,j max If q = 0 Interchange rows i + 1 and j max of A Interchange columns i + 1 and j max of A Interchange columns i + 1 and j max of S

X = (xi j )

⎧ −ak,j /ak,k+1 ⎪ ⎪ ⎨ −1/ak,k+1 xij = 1 ⎪ ⎪ ⎩ 0

;

Y = (yi j )

;

⎧ ⎨ ak,j yij = 1 ⎩ 0

A

←

YAX

S

←

SX

if i = k + 1, j = k + 1 if i = k + 1, j = k + 1 if i = j = k + 1 otherwise

if i = k + 1 if i = j = k + 1 otherwise

else Extract the submatrix formed by rows and columns i + 1 to N of A as H, then solve for G = Danilevkii(H)  S←S

Ii 0

0 G



k←N end If End while The Danilevskii algorithm is known to be among one of the more precise methods for determination of characteristic polynomials and is relatively efficient compared with Leverier’s approach, although the latter is still considered very accurate but slow. A MATLAB function charpoly is available on the book’s webpage for the evaluation of the characteristic polynomial via the Danilevskii method. The program obtains the matrix S such that S−1 AS is in the form of a block triangular matrix [k] given in (C.16). It also yields a set of polynomial coefficients p nk saved in a cell array. Finally, the set of eigenvalues is also available by solving for the roots of the polynomials. A function poly(A) is also available in MATLAB, which is calculated in reverse; that is, the eigenvalues are obtained first, and then the characteristic polynomial is formed.

Appendix C: Additional Details and Fortification for Chapter 3 EXAMPLE C.7.

Given

⎛ ⎜ ⎜ A=⎜ ⎜ ⎝

1 4 1 2 −1

2 5 −2 1 −1

3 0 0 0 1

0 0 0 1 0

0 0 0 2 1

⎞ ⎟ ⎟ ⎟ ⎟ ⎠

then applying the Danilveskii method, we find ⎛ 1 0 0 0 ⎜ −2.75 −0.25 0.25 0 ⎜ 1.5 0.5 −0.1667 0 S=⎜ ⎜ ⎝ 0 0 0 1 0 0 0 −0.5 ⎛ 0 1 0 0 ⎜ 0 0 1 0 ⎜ 6 6 0 S−1 AS = ⎜ ⎜ −39 ⎝ −0.75 −0.25 0.25 0 5.75 1.25 −.5833 −1

0 0 0 0 0.5 0 0 0 1 2

⎞ ⎟ ⎟ ⎟ ⎟ ⎠ ⎞ ⎟ ⎟ ⎟ ⎟ ⎠

and the characteristic polynomial is given by p 1 (s) p 2 (s)

= =

s3 − 6s2 − 6s + 39 p (s) → s2 − 2s + 1

= =

p 1 (s)p 2 (s) s5 − 8s4 + 7s3 + 45s2 − 84s + 39

663

APPENDIX D

Additional Details and Fortification for Chapter 4

D.1 Proofs of Identities of Differential Operators The proofs for the identities of differential operations of orthogonal curvilinear coordinates are given as follows: 1. Gradient (4.90): apply (4.89) on ψ. 2. Divergence (4.91): Using (4.55), ∇ · (wa δa ) = δa · ∇wa + wa ∇ · δa

(D.1)

The first term in (D.1) can be expanded using (4.90)as follows:   1 ∂wa 1 ∂wa 1 ∂wa 1 ∂wa δa · ∇wa = δa · δa + δb + δc = αa ∂a αb ∂b αc ∂c αa ∂a

(D.2)

From (4.87) and (4.88), δa = δb × δc = (αb νb) × (αc νc ) = αbαc ∇b × ∇c Then the second term in (D.1) becomes, wa ∇ · δa

=

wa ∇ · (αbαc ∇b × ∇c)

=

wa (αbαc ∇ · (∇b × ∇c)) + wa (∇b × ∇c) · ∇ (αbαc )   1 1 ∂(αbαc ) 1 ∂(αbαc ) 1 ∂(αbαc ) wa δa · δa + δb + δc αbαc αa ∂a αb ∂b αc ∂c

= =

wa

1 ∂(αbαc ) αa αbαc ∂a

(D.3)

where we used the fact that ∇ · (∇b × ∇c) = 0 (see Exercise E4.17). Substituting (D.2) and (D.3) into (D.1), ∇ · (wa δa ) =

1 ∂wa 1 ∂(αbαc ) 1 ∂(wa αbαc ) + wa = αa ∂a αa αbαc ∂a αa αbαc ∂a

Similarly, we can obtain ∇ · (wbδb) = 664

1 ∂(αa wbαc ) αa αbαc ∂b

;

∇ · (wc δc ) =

1 ∂(αa αbwc ) αa αbαc ∂c

Appendix D: Additional Details and Fortification for Chapter 4

Combining, 1 ∇ ·w= αa αbαc



∂(wa αbαc ) ∂(αa wbαc ) ∂(αa αbwc ) + + ∂a ∂b ∂c



3. Curl (4.92): Using (4.56) and (4.61), the curl of wa δa can be expanded as follows: ∇ × (wa δa )

=

∇ × (wa αa ∇a)

=

wa αa (∇ × ∇a) + ∇ (wa αa ) × ∇a A BC D =0    1 ∂(wa αa ) 1 ∂(wa αa ) 1 ∂(wa αa ) 1 δ + δb + δc δ αa a ∂a αb ∂b αc ∂c αa a

= =

−

1 ∂(wa αa ) 1 ∂(wa αa ) δc + δb αa αb ∂b αa αc ∂c

Similarly, ∇ × (wbδb)

=

1 ∂(wbαb) 1 ∂(wbαb) δc − δa αbαa ∂a αbαc ∂c

∇ × (wc δc )

=

1 ∂(wc αc ) 1 ∂(wc αc ) δa − δb αc αb ∂b αc αa ∂a

Combining all three curls, ⎡

∇ ×w

=

1 ⎢ ⎣ αa δa αa αbαc



∂(αc wc ) ∂(αbwb) − ∂b ∂c 

+ αbδb 

+ αc δc



∂(αa wa ) ∂(αc wc ) − ∂c ∂a ∂(αbwb) ∂(αa wa ) − ∂a ∂b





⎤ ⎥ ⎦

4. Laplacian of scalar fields (4.93): Substituting w = ∇ψ =

1 ∂ψ 1 ∂ψ 1 ∂ψ δa + δb + δc αa ∂a αb ∂b αc ∂c

into (4.91),  '  ( '  ( '  ( ∂ 1 αbαc ∂ψ ∂ αa αc ∂ψ ∂ αa αb ∂ψ ∇ · ∇ψ = + + αa αbαc ∂a αa ∂a ∂b αb ∂b ∂c αc ∂c

665

666

Appendix D: Additional Details and Fortification for Chapter 4

5. Gradient-Vector Dyad (4.94): ⎛ ⎞  ∇w = ∇ ⎝ wk δk ⎠ k=a,b,c

=

   (∇wk ) δk + wk ∇δk k=a,b,c

=





k=a,b,c m=a,b,c

 1 ∂wk δm δk + αm ∂m

 wk ∂δ k δ αm m ∂m

k=a,b,c m=a,b,c

D.2 Derivation of Formulas in Cylindrical Coordinates At a point (r, θ, z), the pair of unit vectors δr and δθ is just the pair of unit vectors δx and δy rotated counter-clockwise by an angle θ, which could be achieved using a rotation operator,1 ⎛ ⎞ cos θ sin θ 0 Rr→c = ⎝− sin θ cos θ 0⎠ (D.4) 0 0 1 Because Rr→c is an orthogonal matrix, ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ δx δx δr δr ⎝δθ ⎠ = Rr→c ⎝δy ⎠ ⇐⇒ ⎝δy ⎠ = RTr→c ⎝δθ ⎠ δz δz δz δz

(D.5)

which is relationship 1 in Table 4.6. We can then apply (D.5) for vector v, ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ δr  δr   δx   T  vr vθ vz ⎝δθ ⎠ = v = vx vy vz ⎝δy ⎠ = vx vy vz Rr→c ⎝δθ ⎠ δz δz δz Comparing both ends of the equations, we have ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ vr vx vx vr ⎝vθ ⎠ = Rr→c ⎝vy ⎠ ⇐⇒ ⎝vy ⎠ = RTr→c ⎝vθ ⎠ vz vz vz vz

(D.6)

which is relationship 2 in Table 4.6. For the relationship between the partial differential operators of the rectangular and the cylindrical coordinate system, the chain rule has to be applied. This yields, ⎛ ⎞ ⎛ ⎞⎛ ⎞ ⎛ ⎞⎛ ∂ ⎞ ∂x ∂y ∂z ∂ ∂ cos θ sin θ 0 ⎜ ∂r ⎟ ⎜ ∂r ⎜ ⎟ ∂r ∂r ⎟ ∂x ⎟ ⎟⎜ ⎜ ⎟ ⎜ ⎟ ⎜ ∂x ⎟ ⎜ ⎟ ⎟⎜ ⎜ ⎜ ⎟ ⎜ ⎟⎜ ⎟ ⎜ ⎟⎜ ∂ ⎟ ⎜ ∂ ⎟ ⎜ ∂x ∂y ∂z ⎟ ⎜ ∂ ⎟ ⎜ ⎟ ⎟ ⎜ ⎜ ⎟ ⎜ ⎟⎜ ⎟ − r sin θ r cos θ 0⎟ ⎜ ⎟ ⎜ ∂y ⎟ ⎜ ∂θ ⎟ = ⎜ ∂θ ∂θ ∂θ ⎟ ⎜ ∂y ⎟ = ⎜ ⎟ ⎜ ⎜ ⎟ ⎜ ⎟⎜ ⎟ ⎜ ⎟ ⎟⎜ ⎜ ⎟ ⎜ ⎟⎜ ⎟ ⎟ ⎟⎜ ⎜ ⎜ ∂ ⎟ ⎜ ∂x ∂y ∂z ⎟ ⎜ ∂ ⎟ ⎜ ∂ 0 0 1⎠ ⎝ ⎟ ⎝ ⎠ ⎝ ⎠⎝ ⎠ ⎝ ⎠ ∂z ∂z ∂z ∂z ∂z ∂z 1

Note that the operator in (D.4) will rotate an input vector clockwise by an angle θ. However, because we are rotating the reference axes, the operator would do the reverse; that is, it rotates the axes counterclockwise.

Appendix D: Additional Details and Fortification for Chapter 4

667

  Let Dr→c = diag 1, r, 1 . Then, ⎛

⎛ ⎞ ⎞ ∂ ∂ ⎜ ∂r ⎟ ⎜ ∂x ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜∂⎟ ⎜∂ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ∂θ ⎟ = Dr→c Rr→c ⎜ ∂y ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜∂⎟ ⎜∂ ⎟ ⎝ ⎠ ⎝ ⎠ ∂z ∂z

⎛

⎛ ⎞ ⎞ ∂ ∂ ⎜ ∂x ⎟ ⎜ ∂r ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜∂ ⎟ ⎜∂⎟ ⎜ ⎟ ⎟ T −1 ⎜ ⎜ ∂y ⎟ = Rr→c Dr→c ⎜ ∂θ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜∂ ⎟ ⎜∂⎟ ⎝ ⎠ ⎝ ⎠ ∂z ∂z

⇐⇒

(D.7)

which is relationship 3 in Table 4.6. To obtain the relationship of the gradient operator ∇ between the rectangular and the cylindrical coordinates, we can apply both (D.5) and (D.7), ⎛

∇

=

 δx

δy

⎛

⎞

⎞⎛

∂ ⎜ ⎜ ∂x ⎟ ⎜ ⎜ ⎟ ⎜ ⎜ ⎟ ⎜ ∂ ⎟ ⎜ ⎜ ⎜ ⎟ ⎜ δz ⎜ ∂y ⎟ = ⎜ δr ⎜ ⎟ ⎜ ⎜ ⎟ ⎜ ⎜∂ ⎟ ⎜ ⎝ ⎠ ⎜ ⎜ ∂z ⎝

δθ

⎞ ⎞ ∂ ⎟ ⎟⎜ ⎟⎜ ⎜ ∂r ⎟⎟ ⎟⎜ ⎜ ⎟⎟ ⎟⎜ ⎜ ⎟⎟ ⎟⎜ ⎜ ∂ ⎟⎟ ⎟⎜ T  ⎟⎟ −1 ⎜ ⎜ ⎟ δz Rr→c ⎟ ⎜Rr→c Dr→c ⎜ ∂θ ⎟⎟ ⎜ ⎟⎟ ⎟⎜ ⎜ ⎟⎟ ⎟⎜ ⎜ ∂ ⎟⎟ ⎟⎜ ⎝ ⎠⎟ ⎟ ⎟⎜ ∂z ⎠ ⎠⎝

⎛

⎛

 =

δx

1 δ r y

⎞ ∂ ⎜ ∂r ⎟ ⎜ ⎟ ⎟ ⎜ ⎜∂⎟ ⎜ ⎟ δz ⎜ ∂θ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜∂⎟ ⎝ ⎠ ∂z

(D.8)

which is relationship 4 in Table 4.6. To obtain the partial derivatives of unit vectors in the cylindrical coordinate systems, note that: 1. The direction and magnitude of δr , δθ , and δz will not change if we just modify the r position. Thus ∂δ ∂δr ∂δ = θ = z =0 ∂r ∂r ∂r 2. Likewise, the direction and magnitude of δr , δθ , and δz will not change if we just modify the z position. Thus ∂δ ∂δr ∂δ = θ = z =0 ∂z ∂z ∂z 3. If we just change the θ position, the direction or magnitude of δz will also not change. Thus ∂δz =0 ∂θ

668

Appendix D: Additional Details and Fortification for Chapter 4

Figure D.1. Unit vectors along r at different θ positions.

What remains is the behavior of δr and δθ as we change the θ position. In both cases, the directions do change. Let us first look at how δr changes with θ. The partial derivative of δr with respect to θ is given by ∂δr δ (r, θ + θ, z) − δr (r, θ, z) = lim r

θ→0 ∂θ

θ where the subtraction is a vector subtraction. This is shown in (the right side of) Figure D.1. As θ → 0, we can see that the vector difference will be pointing perpendicular to δr (r, θ, z). Thus   ∂δr direction = direction (δθ ) ∂θ For the magnitude,      lim δr (r, θ + θ) − δr (r, θ)  = lim 2|δr | sin θ/2 = 1  θ→0  θ→0

θ

θ Because the direction and magnitude matches δθ , ∂δr = δθ ∂θ

(D.9)

Using a similar argument for δθ , ∂δθ δ (r, θ + θ, z) − δθ (r, θ, z) = lim θ

θ→0 ∂θ

θ The vector subtraction is shown in Figure D.2, where the limit yields a vector that is pointing in opposite direction of δr . The magnitude of the limit is also 1. Thus ∂δθ = − δr ∂θ

Figure D.2. Unit vectors along θ at different θ positions.

(D.10)

Appendix D: Additional Details and Fortification for Chapter 4

669

Alternatively, to find the derivatives of the unit vectors of cylindrical coordinates, we could use the fact that δx , δy , and δz have fixed magnitudes and direction. Then using (D.4) and (D.5), ⎛ ⎞ δ ∂ ⎝ r⎠ δθ ∂r δz ⎛ ⎞ δ ∂ ⎝ r⎠ δθ ∂θ δz

=

=

=

= ⎛ ⎞ δ ∂ ⎝ r⎠ δθ ∂z δz

⎛ ⎞ ⎛ ⎞ δr 0 ∂ T ⎝ ⎠ ⎝ Rr→c Rr→c δθ = 0⎠ ∂r δz 0 ⎛ ⎞   δr ∂ Rr→c RTr→c ⎝δθ ⎠ ∂θ δz ⎛ ⎞⎛ − sin θ cos θ 0 cos θ ⎝− cos θ − sin θ 0⎠ ⎝ sin θ 0 0 0 0 ⎛ ⎞ δθ ⎝−δr ⎠ 0 ⎛ ⎞ ⎛ ⎞   δr 0 ∂ Rr→c RTr→c ⎝δθ ⎠ = ⎝0⎠ ∂z δz 0



=



− sin θ cos θ 0

⎞⎛ ⎞ 0 δr 0⎠ ⎝δθ ⎠ 0 δz

D.3 Derivation of Formulas in Spherical Coordinates To transform the unit vectors in rectangular coordinates to spherical coordinates at a point (x, y, z) ↔ (r, θ, φ), we need the following sequence of operations:   1. A rotation of φ radians counterclockwise along the δx , δy plane using the rotation operator Rrs1 .   2. A rotation of θ radians clockwise along the δx , δz plane using the rotation operator Rrs2 . 3. A reordering of the unit vectors using the permutation operator Ers . where, ⎛

Rrs1

cos φ = ⎝− sin φ 0

sin φ cos φ 0

⎞ 0 0⎠ 1

⎛

Rrs2

cos θ =⎝ 0 sin θ

0 1 0

⎞ − sin θ 0 ⎠ cos φ

⎛

0 Ers = ⎝1 0

0 0 1

⎞ 1 0⎠ 0

Combining all three orthogonal operators sequence will yield an  in theprescribed   orthogonal operator used to transform δx , δy , δz to δr , δθ , δφ : ⎛

Rr→s = Ers Rrs2 Rrs1

sin θ cos φ = ⎝cos θ cos φ − sin φ

sin θ sin φ cos θ sin φ cos φ

⎞ cos θ − sin θ⎠ 0

(D.11)

670

Appendix D: Additional Details and Fortification for Chapter 4

Then, following the same approach used during transformations between rectangular and cylindrical coordinates, we have ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ δx δx δr δr T ⎝ δθ ⎠ = Rr→s ⎝δy ⎠ ⎝ ⎠ ⎝ δy = Rr→s δθ ⎠ ⇐⇒ (D.12) δφ δφ δz δz ⎛ ⎞ ⎛ ⎞ vx vr ⎝ vθ ⎠ = Rr→s ⎝vy ⎠ vφ vz

⇐⇒

⎛ ⎞ ⎛ ⎞ vx vr ⎝vy ⎠ = RTr→s ⎝ vθ ⎠ vz vφ

(D.13)

The partial differential operators between the rectangular and spherical coordinate system are obtained by using the chain rule, ⎛ ⎞ ⎛ ⎞⎛ ⎞ ⎛ ⎞⎛ ∂ ⎞ ∂ ∂x ∂y ∂z ∂ sθ cφ sθ sφ cθ ⎜ ∂r ⎟ ⎜ ∂r ⎜ ⎟ ∂r ∂r ⎟ ∂x ⎟ ⎟⎜ ⎜ ⎟ ⎜ ⎟ ⎜ ∂x ⎟ ⎜ ⎟ ⎜ ⎟⎜ ⎜ ⎟ ⎜ ⎜ ⎟⎜ ⎟ ⎜ ⎟ ⎟ ⎜ ∂ ⎟ ⎜ ∂x ⎜ ⎜ ⎟ ⎟ ∂y ∂z ∂ ⎟⎜ ∂ ⎟ ⎜ ⎟ ⎜ ⎟⎜ ⎟ ⎜ ⎟ rc c rc s −rs θ φ θ φ θ⎟ ⎜ ∂θ ⎟ = ⎜ ∂θ ⎜ ⎟=⎜ ∂θ ∂θ ⎟ ∂y ⎟ ⎟⎜ ⎜ ⎟ ⎜ ⎜ ⎟ ⎜ ∂y ⎟ ⎜ ⎟⎜ ⎟ ⎜ ⎟ ⎜ ⎟⎜ ⎟ ⎜ ⎟ ∂ ⎟ ⎜ ∂ ⎟ ⎜ ∂x ⎟ ∂y ∂z ⎟ ⎜ ∂ ⎟ ⎜ 0 ⎠⎜ ⎝ ⎠ ⎝ ⎝ ⎠ ⎠ ⎝ ⎠ ⎝ − rsθ sφ rsθ cφ ∂φ ∂φ ∂φ ∂φ ∂z ∂z   Let Dr→s = diag 1, r, r sin θ . Then, ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ ∂ ∂ ∂ ∂ ⎜ ∂r ⎟ ⎜ ∂x ⎟ ⎜ ∂x ⎟ ⎜ ∂r ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ∂ ⎟ ⎜∂ ⎟ ⎜∂ ⎟ ⎜ ∂ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ∂θ ⎟ = Dr→s Rr→s ⎜ ∂y ⎟ ⇐⇒ ⎜ ∂y ⎟ = RTr→s D−1 r→s ⎜ ∂θ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ∂ ⎟ ⎜∂ ⎟ ⎜∂ ⎟ ⎜ ∂ ⎟ ⎝ ⎠ ⎝ ⎠ ⎝ ⎠ ⎝ ⎠ ∂φ ∂z ∂z ∂φ

(D.14)

To obtain the relationship of the gradient operator ∇ between the rectangular and the spherical coordinates, we can apply both (D.12) and (D.14), ⎛ ⎞ ⎞⎛ ⎛ ⎞ ⎛ ⎞ ∂ ∂ ⎟ ⎜ ⎟⎜ ⎟⎜ ⎜ ∂x ⎟ ⎜ ⎜ ∂r ⎟⎟ ⎜ ⎜ ⎟ ⎜ ⎟ ⎜ ⎜ ⎟⎟ ⎟⎜ ⎜ ⎟ ⎜ ⎜ ⎟⎟ ⎟⎜ ⎜ ∂ ⎟⎟ ∂ ⎟ ⎜ ⎜ ⎟  ⎜  ⎜ ⎟ ⎟⎟ T −1 ⎜ ⎜ ⎟ ⎟ δx δy δz ⎜ ∂y ⎟ = ⎜ δ δ δ R ∇ = R D r→s ⎟ ⎜ r→s r→s ⎜ ∂θ ⎟⎟ r θ φ ⎜ ⎜ ⎟ ⎜ ⎜ ⎟⎟ ⎟⎜ ⎜ ⎟ ⎜ ⎜ ⎟ ⎟⎜ ⎜∂ ⎟ ⎜ ⎜ ∂ ⎟⎟ ⎟⎜ ⎝ ⎠ ⎜ ⎝ ⎠⎟ ⎟ ⎟⎜ ∂z ∂φ ⎠ ⎝ ⎠⎝ ⎛

=

 δx

1 δ r y

⎞ ∂ ⎜ ∂r ⎟ ⎜ ⎟ ⎟ ⎜ ⎜ ∂ ⎟ 1 ⎜ ⎟ δ ⎜ ∂θ ⎟ ⎟ r sin θ z ⎜ ⎜ ⎟ ⎜ ∂ ⎟ ⎝ ⎠ ∂φ

(D.15)

Appendix D: Additional Details and Fortification for Chapter 4

Figure D.3. Unit vectors at fixed r and φ. The unit vectors are represented by: a = δr (r, θ, φ), b = δθ (r, θ, φ), c = δr (r, θ + θ, φ), d = δθ (r, θ + θ, φ).

To obtain the partial derivatives of unit vectors in the spherical coordinate systems, note that: 1. The direction and magnitude of δr , δθ , and δφ will not change if we just modify the r position. Thus ∂δφ ∂δr ∂δ = θ = =0 ∂r ∂r ∂r 2. The direction and magnitude of δφ will not change if we just modify the θ position. Thus ∂δφ ∂θ

=0

The remaining partial derivatives of unit vectors will change their direction based on their position in space. For a fixed r and φ, the vector subtractions are shown in Figure D.3, and the partial derivatives are then given by ∂δr = δθ ∂θ

∂δθ = − δr ∂θ

(D.16)

For a fixed r and θ, the vector subtractions are shown in Figure D.4. Note that four of the unit vectors are first projected into the horizontal plane prior to taking limits. The partial derivatives are then given by: ∂δφ ∂φ

= − cos θδθ − sin θδr ;

∂δr ∂δθ = sin θδφ ; = cos θδφ ∂φ ∂φ

(D.17)

671

672

Appendix D: Additional Details and Fortification for Chapter 4

Figure D.4. Unit vectors at fixed r and θ. The unit vectors are represented by: a = δr (r, θ, φ), b = δθ (r, θ, φ), c = δφ (r, φ), d = δr (r, θ, φ + φ), f = δθ (r, θ, φ + φ), g = δφ (r, θ, φ + φ). The unit vectors projected into the horizontal planes are: & a = δr (r, θ, φ) sin θ, & b= δθ (r, θ, φ) cos θ, & d = δr (r, θ, φ + φ) sin θ, & f = δθ (r, θ, φ + φ) cos θ.

Alternatively, to find the derivatives of the unit vectors of spherical coordinates, we could use the fact that δx , δy , and δz have fixed magnitudes and direction. Then using (D.11) and (D.12), ⎛ ⎞ ⎛ ⎞ ⎛ ⎞   δ δr 0 ∂ ⎝ r⎠ ∂ T ⎝ δθ = Rr→s Rr→s δθ ⎠ = ⎝0⎠ ∂r ∂r δφ δφ 0 ⎛ ⎞ ⎛ ⎞   δ δr ∂ ⎝ r⎠ ∂ δθ = Rr→s RTr→s ⎝ δθ ⎠ ∂θ ∂θ δφ δφ ⎞⎛ ⎞⎛ ⎞ ⎛ sθ cφ cθ cφ −sφ δr cθ cφ cθ sφ −sθ = ⎝−sθ cφ −sθ sφ −cθ ⎠ ⎝sθ sφ cθ sφ cφ ⎠ ⎝δθ ⎠ 0 0 0 cθ −sθ 0 δz ⎛ ⎞ δθ = ⎝−δr ⎠ 0 ⎛ ⎞ ⎛ ⎞   δ δr ∂ ⎝ r⎠ ∂ δθ = Rr→s RTr→s ⎝ δθ ⎠ ∂φ ∂r δφ δφ ⎞⎛ ⎞⎛ ⎞ ⎛ sθ cφ cθ cφ −sφ δr −sθ sφ sθ cφ 0 = ⎝−cθ sφ cθ cφ 0⎠ ⎝sθ sφ cθ sφ cφ ⎠ ⎝δθ ⎠ −sφ 0 cθ −sθ 0 −cφ δz ⎞⎛ ⎞ ⎛ δr 0 0 sθ = ⎝ 0 0 cθ ⎠ ⎝ δθ ⎠ δφ −sθ −cθ 0

APPENDIX E

Additional Details and Fortification for Chapter 5

E.1 Line Integrals Line integrals are generalizations of the ordinary integrals of single-variable functions to handle cases in which variations occur along specified curves in two or three dimensions. The line integrals therefore consists of three components: the path of integration C(x, y, z), which is a continuous curve; the integrand F (x, y, z), which is a scalar function; and the differential dλ. Definition E.1. A line integral of F (x, y, z), with respect to variable λ and path C(x, y, z), is defined by  F (x, y, z) dλ = C

lim

λi →0,N→∞

N 

F (xi , yi , zi ) λi

(E.1)

i=0

In most applications, the differential dλ is set to either dx, dy, dz or ds, where " ds = dx2 + dy2 + dz2 (E.2) For the 2D case, F = F (x, y) and the path C = C(x, y). Figure E.1 gives the area interpretation of the line integrals. The integral C F (x, y)ds is the area under the curve F (x, y)as the point travels along curve C. Conversely, the line integral with respect to  x, C F (x, y)dx is the area projected onto the plane y = 0. The projected integral C Fdx is with respect to segments where C(x, y) has to be single-valued with respect to x. Otherwise, the integration path will have to be partitioned into segments such that it is single-valued with respect to x. For example, the integration path from A to B in Figure E.2 will have to be partitioned into segment ADE, segment EF , and segment FGB. Thus for the integration path shown in Figure E.2, the line integral with respect to x is given by     F (x, y)dx = F (x, y)dx + F (x, y)dx + F (x, y)dx (E.3) C

[ADE]

[EF ]

[FGB]

For the 3D case, another interpretation is more appropriate. One could visualize a mining activity that accumulates substance, say, Q, along path C in the ground containing a concentration distribution of Q. Let F (x, y, z) be the amount of Q 673

674

Appendix E: Additional Details and Fortification for Chapter 5

Figure E.1. Area interpretation of line integrals.

gathered per unit length traveled. Then, along the differential path ds, an amount F (x, y, z)ds will have been accumulated, and the total amount gathered along the path C becomes C F (x, y, z)ds. Conversely, the integral C F (x, y, z)dx is the amount of Q gathered  along the projected path in the x-direction. In this mining scenario, the line integral C F (x, y, z)dx does not appear to be as relevant compared with the line integral with respect to s. However, these line integrals are quite useful during the computation of surface integrals and volume integrals because differential surfaces s are often described by dx dy, dx dz, or dy dz, and differential volumes are often described by the product dx dy dz.1 Another example is when the integral involves the position vector r of the form     f · dr = f x dx + f y dy + f zdz C

C

E.1.1 The Path of Integration The path of integration will be assumed to be a continuous and sectionally smooth curve. The curve can either be open or closed. A path is closed if the starting point of the path coincides with the end point of the path. Otherwise, the path is said to be open. In either case, the direction of the path is crucial during integration. If the path is not self-intersecting at points other than the terminal points, then we say that the curve is a simple curve. Non-simple curves can be treated as the direct sum of simple curves, as shown in Figure E.3. When the path is closed and non-intersecting, we often indicate a closed path by the following notation: 3 Fds C is a closed, sectionally smooth, nonintersecting path C

  A 3D path can be described generally by C = x(t), y(t), z(t) = r(t), where r is the position vector and t is a parameter going from t = 0 to t = 1.2 In some cases, the curve can be parameterized by either x = t, y = t or z = t. In these cases, the other variables are said to possess an explicit form, for example, for x = t, we can use y = y(x) and z = z(x).3 1 2 3

One could then expect that in other coordinate systems, dλ may need involve those coordinates, for example, dr, dθ, dφ, and so forth. A more general formulation would be to let the parameter start at t = a and end with t = b, where b > a. Using translation and scaling, this case could be reduced back to a = 0 and b = 1. The parameterizations can also originate from coordinate transformations such as polar, cylindrical, or spherical coordinates.

Appendix E: Additional Details and Fortification for Chapter 5

Figure E.2. A curve in which the projection of C onto x or y is not single valued.

EXAMPLE E.1.

675

A

Consider the closed elliptical path given described by 

x+3 2

2 + (y + 2)2 = 4

(E.4)

traversed in the counterclockwise direction as shown in Figure E.4. Let the path start at point a : (−7, −2) and pass through points b : (−3, −4), c : (1, −2), d : (−3, 0), and then back to a. The path can then be described in three equivalent ways: 1. Parameterized Form. Path Cabcda : x

=

−3 − 4 cos(2πt)

y

=

−2 − 2 sin(2πt) from t = 0 to t = 1

2. Explicit function of x. Path Cabcda = Cabc + Ccda where

6 Cabc : y = −2 −

 4−

6 Ccda : y = −2 +

4−



x+3 2 x+3 2

2 from x = −7 to x = 1 2 from x = 1 to x = −7

3. Explicit function of y. Path Cabcda = Cab + Cbcd + Cda

Figure E.3. Separation into simple curves.

676

Appendix E: Additional Details and Fortification for Chapter 5

1

d 0

y

−1

−2

a

c

Figure E.4. A close path of integration in counterclockwise direction.

−3

−4

b −5

−8

−6

−4

−2

0

2

x

where

" Cab : x = −3 − 2 4 − (y + 2)2

from y = −2 to y = −4

" : x = −3 + 2 4 − (y + 2)2

from y = −4 to y = 0

" Cda : x = −3 − 2 4 − (y + 2)2

from y = 0 to y = −2

Cbcd

E.1.2 Computation of Line Integrals With the parameterized form of path C based on t, the integrand also becomes a function of t, that is,   F x(t), y(t), z(t) = g(t) (E.5) Using the chain rule, the line integrals become 

  dx g(t) dt dt 0   1 dy g(t) dt dt 0   1 dz g(t) dt dt 0 ⎛ ⎞ 6  2  2  2  1 dx dy dz ⎝g(t) ⎠ dt + + dt dt dt 0 

F (x, y, z)dx

=

F (x, y, z)dy

=

F (x, y, z)dz

=

F (x, y, z)ds

=

C



C



C

 C

1

(E.6)

However, if an explicit form is possible, these should be attempted in case they yield simpler calculations. For instance, suppose y = y(x) and z = z(x); then setting

Appendix E: Additional Details and Fortification for Chapter 5

x = t, (E.6) are modified by replacing dx/dt = 1, dy/dt = dy/dx and dz/dt = dz/dx with the lower limit xstart and upper limit xend . For example,  xend  F (x, y, z)dx = F (x, y(x), z(x))dx C

xstart

Consider the scalar function given by

EXAMPLE E.2.

F (x, y) = 2x + y + 3 and the counter-clockwise elliptical path of integration given in Example E.1. Using the parameterized form based on t, x(t)

=

−3 − 4 cos (2πt)

y(t)

=

g(t)

=

−2 − 2 sin (2πt)   F x(t), y(t) = 2 (−3 − 4 cos (2πt)) + (−2 − 2 sin (2πt)) + 3

and

Thus  F (x, y)dx

dx

=

8π sin (2πt) dt

dy

=

ds

=

−4π cos (2πt) dt " 4π 4 sin2 (2πt) + cos2 (2πt)dt

 =

C

1

g(t) (8π sin (2πt)) dt = −8π

0



 F (x, y)dy

=

C

1

0



 F (x, y)ds

=

C

1

g(t) (−4π cos (2πt)) dt = 16π   " g(t) 4π 4 sin2 (2πt) + cos2 (2πt) dt = −96.885

0

Using the explicit form y = y(x) for the integration path

Cabc

Ccda

C = Cabc + Ccda 6   x+3 2 : y = yabc = −2 − 4 − 2 6   x+3 2 : y = ycda = −2 + 4 − 2

from x = −7 to x = 1 from x = 1 to x = −7

The integrand and differentials for the subpaths are ⎛ ⎞ 6 2  x + 3 ⎠ F (x, y)abc = 2x + 3 + ⎝−2 − 4 − 2 ⎛ F (x, y)cda

=

2x + 3 + ⎝−2 +

6 4−



x+3 2

2

⎞ ⎠

677

678

Appendix E: Additional Details and Fortification for Chapter 5

   

dy dx dy dx ds dx ds dx

 = 

abc

=

x+3 + ! 2 (1 − x) (x + 7) " 1 + dy2abc

=

" − 1 + dy2cda

= cda

 

x+3 − ! 2 (1 − x) (x + 7)

abc

cda

Note that ds has a negative sign for the subpath [cda]. This is because the direction of ds is opposite that of dx in this region. The line integrals are then given by  1  −7  F (x, y)dx = F (x, y)abc dx + F (x, y)cda dx −7

C

= 

−8π 

F (x, y)dy

=

F (x, y)ds

=

C

=

F (x, y)abc

dy dx



 dx +



−7

F (x, y)cda 1

abc

dy dx

 dx cda

16π 





1 −7

C

=

1



1 −7

F (x, y)abc

ds dx



 dx +



−7

F (x, y)cda 1

abc

ds dx

 dx cda

−96.885

This shows that either the parameterized form or the explicit form approach can be used to obtain the same values. The choice is usually determined by the tradeoffs between the complexity of the parameterization procedure and the complexity of the resulting integral.

E.2 Surface Integrals Definition E.2. A surface integral of F (x, y, z), with respect to area A and surface of integration S(x, y, z), is defined by  F (x, y, z) dA = S

lim

Ai →0,N→∞

N 

F (xi , yi , zi ) Ai

(E.7)

i=0

In most applications, the differential area is specified for either dA = dx dy, dydz, dx dz, or dS, where dS is the differential area of the surface of integration To visualize surface integrals, we could go back to the mining scenario for the substance Q, except now the accumulation  is obtained by traversing a surface instead of a path. Thus the surface integral S f (x, y, z)dS can be thought of as the total amount mined by sweeping the total surface area S.

Appendix E: Additional Details and Fortification for Chapter 5

679

E.2.1 Surface of Integration A general parametric description of surface is based on two independent parameters, u and v,   S : x(u, v), y(u, v), z(u, v) as u and v vary independently in a closed domain. (E.8) If the parameterization can be done by letting u = x and v = y, then the surface is given by the explicit form for z S : z = z(x, y)

as x and y vary independently in a closed domain

(E.9)

Other explicit forms are possible, for example, y = y(x, z) and x = x(y, z). Two important variables are needed during the calculation of surface integrals:  , and the differential area dS at the point (x, y, z). As the unit normal vector n discussed in Section 4.6, the unit normal to a surface is given by (4.30), that is, = n

tu × tv

(E.10)

tu × tv

where tu = Specifically, we have tu × tv =



∂r ∂u

and tv =

∂r ∂v

∂(y, z) ∂(z, x) ∂(x, y) δ + δ + δ ∂(u, v) x ∂(u, v) y ∂(u, v) z

 (E.11)

where we used the shorthand notation for the Jacobian determinants given by ⎛ ⎞ ∂a ∂a ⎜ ∂c ∂d ⎟ ∂(a, b) ⎟ = det⎜ ⎝ ∂b ∂b ⎠ ∂(c, d) ∂c ∂d However, the differential surface area is given by the area of the parallelogram formed by differential arcs form by movement along constant v and u, respectively, that is, the area formed by tu du and tv dv. Thus     tu du × tv dv = tu × tv du dv dS = 6       ∂(y, z) 2 ∂(z, x) 2 ∂(x, y) 2 = + + du dv (E.12) ∂(u, v) ∂(u, v) ∂(u, v) If the explicit form z = z(x, y) is possible, that is, with x = u and y = v, the formulas reduce to the more familiar ones, that is,   ∂z ∂z δx − δy + δz ∂x ∂y n = 6 (E.13)  2  2 ∂z ∂z 1+ + ∂x ∂y

680

Appendix E: Additional Details and Fortification for Chapter 5

Figure E.5. The boundary of domain D for parameter space can either have (a) an independent range, or (b) an interdependent range.

⎛6 dS

=

⎝ 1+



∂z ∂x

2

 +

∂z ∂y

2

⎞ ⎠ dxdy

(E.14)

Note that with the square root, the choice for sign depends on the interpretation of the surface direction. In most application, for a surface that encloses a region of 3D space, the surface outward of the enclosed region is often given a positive sign.

Consider a circular cylinder of radius R of height h with the bottom base centered at the origin. The differential area at the top and the bottom can be parameterized in terms of r and θ; that is, x = r cos θ and y = r sin θ. At the top, we have z = 0 and set u = r and v = θ as the parameterization. At the bottom, we have z = h but will need to set u = r and v = θ as the parameterization to obtain the expected outward normal direction. Thus, for the top, EXAMPLE E.3.

⎛

δx tu × tv = det⎝ cos θ −r sin θ

δy sin θ r cos θ

For the bottom, we have ⎛ δx δy tu × tv = det⎝ −r sin θ r cos θ cos θ sin θ

⎞ δz 0 ⎠ = rδz 0

→

⎞ δz 0 ⎠ = −rδz 0

→

 top n

=

δz

dStop

=

r dr dθ

 bottom n

=

−δz

dSbottom

=

r dr dθ

For the side of the cylinder, we let u = θ and v = z and r = R. Then ⎛ ⎞ δx δy δz   tu × tv = det⎝ −R sin θ R cos θ 0 ⎠ = R cos θδx + sin θδy = Rδr 0 0 1 which the yields  side = δr n

and

dSside = Rdθdz

E.2.2 Computation of Surface Integrals Under the parameterized form of the surface of integration, the domain of the parameter space is a closed 2D plane in the (u, v) space. The boundary may either be defined independently by fixed ranges for u and v, or the boundary has to described by giving explicit dependencies of u on v or vice versa (see Figure E.5).

Appendix E: Additional Details and Fortification for Chapter 5

681

Figure E.6. The two possible domain descriptions: (a) boundary is partitioned into two segments such that v = φ(u), and (b) boundary is partitioned into two segments such that u = ψ(v) .

If the ranges of u and v are independent, then domain D can, without loss of generality, be given as D : 0≤u≤1 ;

0≤v≤1

The surface integral becomes 



1

F (x, y, z)dS = S



0

1

g(u, v)du dv

0

where 6 g(u, v) = f (x(u, v), y(u, v), z(u, v))



∂(y, z) ∂(u, v)



2 +

∂(z, x) ∂(u, v)



2 +

∂(x, y) ∂(u, v)

2

(E.15) Thus  h(v)

=

1

g(u, v)du

holding v constant

0



 F (x, y, z)dS S

=

1

h(v)dv

(E.16)

0

If u and v are interdependent at the boundary of the parameter space, then two domain descriptions are possible: Du : ulower ≤ u ≤ uupper

;

φ0 (u) ≤ v ≤ φ1 (u)

(E.17)

ψ0 (v) ≤ u ≤ ψ1 (v)

(E.18)

or Dv : vlower ≤ v ≤ vupper

;

where ulower , uupper , vlower and vupper are constants. Both domain descriptions are shown in Figure E.6, and both are equally valid.

682

Appendix E: Additional Details and Fortification for Chapter 5

With the first description given by (E.17), the surface integral is given by  φ1 (u) g(u, v)dv holding u constant h(u) = φ0 (u)



 =

F (x, y, z)dS

uupper

h(u)du

S

(E.19)

ulower

where g(u, v) is the same function as in (E.15). Similarly, using the second description given in (E.18),  ψ1 (v) g(u, v)du holding u constant h(v) = ψ0 (v)



 =

F (x, y, z)dS

vupper

vlower

S

h(v)dv

(E.20)

For the special case in which the surface is given by z = z(x, y), u

=

v=y 6

x



g(u, v)

=

ulower

=

 2  ∂z 2 ∂z g(x, y) = f (x, y, z(x, y)) 1 + + ∂x ∂y xlower uupper = xupper

φ1 (u)

=

φ1 (x)

φ0 (u) = φ0 (x)

vlower

=

ylower

vupper = yupper

ψ1 (v)

=

ψ1 (y)

ψ0 (v) = ψ0 (y)

EXAMPLE E.4.

Consider the integrand given by F (x, y, z) = 2x + y − z + 3

and the surface of integration provided by the ellipsoid,  y 2 x2 + + z2 = 1 2 A parameterized form is given by x = sin(u) cos(v) ;

y = sin(u) sin(v) ;

where the parameter domain is described by 0 ≤ u ≤ 2π

0≤v≤π

The Jacobian determinants can be evaluated as ∂(x, y) = −2 sin(u) cos(u) ∂(u, v) ∂(y, z) ∂(u, v)

=

−2 sin2 (u) cos(v)

∂(x, z) ∂(u, v)

=

sin2 (u) cos(v)

z = cos(u)

Appendix E: Additional Details and Fortification for Chapter 5

which then gives 6 g(u, v)



∂(y, z) ∂(u, v)

=

F (x, y, z)

=

α(u, v)β(u, v)

2

 +

∂(z, x) ∂(u, v)

2

 +

∂(x, y) ∂(u, v)

2

where α(u, v) = 2 sin(u) cos(v) + 2 sin(u) sin(v) − cos(u) + 3 7   β(u.v) =

3 cos2 (v) (cos(u) − 1)2 (cos(u) + 1)2 + (1 + 2 cos2 (u) − 3 cos4 (u))

The surface integral can then be solved numerically to be 

π



0

2π

g(u, v)du dv = 64.4

0

As an alternative, we can partition the elliptical surface into two halves. The upper half and lower half can be described by zu and z , respectively, where 6 zu =



1 − x2 −

y2 2

6



z = − 1 − x2 −



y2 2



In either half, the (x, y)-domain can be described by D : −1≤x≤1

! ! − 2 1 − x2 ≤ y ≤ 2 1 − x2

For the upper half, dzu −2x =! dx 4 − 4x2 − y2

dzu −y/2 =! dy 4 − 4x2 − y2

with an integrand 

6

g u (x, y) = 2x + y −

 1 − x2 −

⎞ ⎛ 6  2 − 16 1 y2 3y ⎠ +3 ⎝ 2 2 −4 + 4x2 + y2

For the lower half, dz 2x =! dx 4 − 4x2 − y2

dz y/2 =! dy 4 − 4x2 − y2

with an integrand  g  (x, y) = 2x + y +

6

 1 − x2 −

⎞ ⎛ 6  2 − 16 1 y2 3y ⎠ +3 ⎝ 2 2 −4 + 4x2 + y2

683

684

Appendix E: Additional Details and Fortification for Chapter 5

Combining everything, we can calculate the surface integral via numerical integration to be  Iu

=

I

=

fdS

=

  S

+1



√ 2 1−x2

√ −2 1−x2 √ +1  2 1−x2

−1

√ −2 1−x2

−1

g u (x, y)dydx = 26.6 g  (x, y)dydx = 37.8

Iu + I = 64.4

which is the same value as the previous answer using the parameterized description.

Remark: In the example just shown, we have used numerical integration. This is usually the preferred route when the integrand becomes too complicated to integrate analytically. There are several ways in which the numerical approximation can be achieved, including the rectangular or trapezoidal approximations or Simpson’s methods. We have also included another efficient numerical integration technique called the Gauss-Legendre quadrature method in the appendix as Section E.4.

E.3 Volume Integrals Definition E.3. A volume integral of F (x, y, z), with respect to W and volume of integration V (x, y, z), is defined by  F (x, y, z) dW = V

lim

Wi →0,N→∞

N 

F (xi , yi , zi ) Wi

(E.21)

i=0

In most applications, the differential volume is specified by dW = dx dy dz. To continue the visual interpretation via mining used earlier for both the line and  surface integrals, the mining activity now accumulates substance Q indicated by V F (x, y, z)dV by carving out a volume V specified by the boundary.

E.3.1 Volume of Integration In most cases, the rectangular coordinate system is sufficient to describe the surface of the volume, and thus the differential volume is given by dV = dx dy dz. However, in other cases, another set of coordinates allow for easier computation, for example, cylindrical or spherical coordinates. Let this set of new coordinates be given by parameters (u, v, w). Let r be the position vector. At a point p, we can trace paths C1 , C2 , and C3 that pass through point p, each path formed by holding the other two parameters fixed. This is shown in Figure E.7, where the differential arcs along each of each curve are given by a, b, and c where a=

∂r du ∂u

;

b=

∂r dv ; ∂v

c=

∂r dw ∂w

Appendix E: Additional Details and Fortification for Chapter 5

685

Figure E.7. Graphical representation of differential volume, dV , as function of u, v, and w. Note that the position described by r can be anywhere on or inside V .

The differential volume is then formed by the formed by a, b, and c, that is,  ⎛  ∂x  ⎜ ∂u  ⎜  ⎜ ∂y    ⎜       dV =  c · (a × b)  = det ⎜ ⎜ ∂u ⎜  ⎜ ∂z   ⎝  ∂u 

absolute value of the triple product ∂x ∂v ∂y ∂v ∂z ∂v

∂x ∂w ∂y ∂w ∂z ∂w

⎞   ⎟ ⎟ ⎟ ⎟ ⎟ du dv dw ⎟ ⎟ ⎟ ⎠  

(E.22)

EXAMPLE E.5. For the spherical coordinates, using x = r sin θ cos φ, y = r sin θ sin φ, and z = r cos θ with the parameters u = r, v = θ, and w = φ, we have ⎞ ⎛ sin θ cos φ r cos θ cos φ −r sin θ sin φ ⎟ ⎜ r sin θ cos φ ⎠dr dθ dφ = r2 sin θ dr dθ dφ dV = det ⎝ sin θ sin φ r cos θ sin φ

cos θ

−r sin θ

0

E.3.2 Computation of Volume Integrals Having determined the differential volume and the integrand, one needs to identify the limits of integration in each of the variables x, y, and z, or of parameters u, v, and w. If the limits are independent, umin ≤ u ≤ umax ; vmin ≤ v ≤ vmax ; wmin ≤ w ≤ wmax the volume integral can be integrated in a nested fashion, ⎛ ⎞ ⎞ ⎛    wmax  vmax  umax   ∂(x, y, z)  ⎟ ⎟ ⎜ ⎜ ⎟   ⎜ FdV = G v, w) (u, ⎝  ∂(u, v, w)  du⎠ dv⎠ dw ⎝ V wmin vmin umin

(E.23)

686

Appendix E: Additional Details and Fortification for Chapter 5

wmin ≤ w ≤ wmax

Figure E.8. A nested description of volume boundaries.

where

  G(u, v, w) = F x(u, v, w), y(u, v, w), z(u, v, w)

(E.24)

If the surface of the volume space is represented by a set of interdependent parameters, there are six possible descriptions that can be used based on the sequence of dependencies. We only describe the sequence w → v → u. As shown in Figure E.8, we can first identify the maximum and minimum value of w, that is, wmin ≤ w ≤ wmax Taking a slice of the volume at a fixed w, a closed region whose boundary can be identified by ηmin (w) ≤ v ≤ ηmax (w) Finally, as shown in Figure E.8, the limits of v for this slice will divide the closed curve into two segments. Each of these segments can then be described by functions of v and w, where the value of w was that used to obtain the slice, ξmin (v, w) ≤ u ≤ ξmax (v, w) Thus we end up with a slightly different nested integration given by     wmax  ηmax (w)  ξmax (v,w)  ∂(x, y, z)    du dvdw (E.25) FdV = G(u, v, w)  ∂(u, v, w)  V wmin ηmin (w) ξmin (v,w) where G(u, v, w) is the same function as in (E.24). EXAMPLE E.6.

Consider the integrand given by F (x, y, z) = 2x + y − z + 3

and the volume of integration given by the ellipsoid  y 2 x2 + + z2 ≤ 1 2 Using the parameterization x = u sin(v) cos(w) ; y = 2u sin(v) sin(w) ; z = u cos(v)

Appendix E: Additional Details and Fortification for Chapter 5

687

with boundaries, 0 ≤ u ≤ 1 ; 0 ≤ v ≤ 2π ; 0 ≤ w ≤ π Let sw = sin(w), cw = cos(w), sv = sin(v), and cv = cos(v). Then the differential volume is    ⎛ ⎞   s c uc c −us s v w v w v w   ⎜  ⎟ ⎜ 2sv sw 2ucv sw 2usv cw ⎟  dV = det ⎜ ⎟ du dvdw  ⎝ ⎠   cv −usv 0     2u2 |sv | du dvdw

=

while the integrand becomes G = 2usv (cw + sw ) − ucv + 3 Combining all the elements together, we can compute the volume integral as  π  2π  1   G(u, v, w) 2u2 |sv | du dvdw = 8π 0

0

0

Alternatively, we could use the original variables x, y and z. Doing so, the differential volume is dV = dx dy dz, whereas the boundary of the volume of integration is given by 7 7  y 2  y 2 Surface boundary: − 1 − z2 − ≤ x ≤ 1 − z2 − 2 2 ! ! −2 1 − z2 ≤ y ≤ 2 1 − z2 −1 ≤

z

≤1

Thus the volume integral is given by   √  √ 1

2 1−z2

−1

√ −2 1−z2

1−z2 −(y/2)2

√

−

1−z2 −(y/2)2

(2x + y − z) dx dy dz = 8π

which is the same answer obtained by using the parameterized description.

E.4 Gauss-Legendre Quadrature The n-point Gauss-Legendre quadrature is a numerical approximation of the integral  +1 −1 F (x)dx that satisfies two conditions: 1. The integral is approximated by a linear combination of n values of F (x), each evaluated at −1 ≤ xi ≤ 1, that is, 

1

−1

and

F (x)dx ≈

n  i=1

Wi F (xi )

(E.26)

688

Appendix E: Additional Details and Fortification for Chapter 5

2. When F (x) is a (2n − 1)th order polynomial, the approximation becomes an  m equality, that is, if F (x) = 2n−1 m=0 am x , 

2n−1 

1

−1

 dx =

m

am x

m=0

n 

Wi

2n−1 

i=1

 am xm i

(E.27)

m=0

Approximations having the form given in (E.26) are generally called quadrature formulas. Other quadrature formulas include Newton-Cotes’ formulas, Simpson’s formulas, and trapezoidal formulas. The conditions given in (E.27) distinguish the values found for Wi and xi as being Gauss-Legendre quadrature parameters. A direct approach to determine Wi and xi is obtained by generating the required equations using (E.27): 

2n−1 

1

−1 2n−1 

2n−1 

1 −1

m=0

am x

dx

=

m=0

am xm

xm dx −

n 

Wi

i=1



1

−1

m=0

 am

 m

n 

xm dx

=

m=0

 Wi xm i

2n−1 

=

2n−1 

 am xm i

m=0

am

n 

Wi xm i

i=1

0

(E.28)

i=1

Because the condition in (E.27) should be true for any polynomial of order (2n − 1), (E.28) should be true for arbitrary values of am , m = 0, 1, . . . , (2n − 1). This yields n 

Wi xm i = γm

for m = 0, 1, . . . , (2n − 1)

(E.29)

i=1

where  γm =

1 −1

xm dx =

⎧ ⎨ 2/(m + 1)

if m is even

⎩

if m is odd

(E.30) 0

This means that we have 2n independent equations that can be used to solve the 2n unknowns: xi and Wi . Unfortunately, the equation becomes increasingly difficult to solve as n gets larger. This is due to the nonlinear terms such as Wi xm i appearing in (E.29). An alternative approach is to separate the problem of identifying the xi values from the problem of identifying the Wi values. To do so, we use Legendre polynomials and take advantage of their orthogonality properties. We first present some preliminary formulas: 1. Any polynomial of finite order can be represented in terms of Legendre polynomials, that is, q  i=0

ci xi =

q  j =0

bj P j (x)

(E.31)

Appendix E: Additional Details and Fortification for Chapter 5

689

where P j (x) is the Legendre polynomial of order j . (To obtain a Legendre polynomial, one can either use definition given in (I.31) or use Rodriguez’s formula given in (9.46).) 2. Let R(2n−1) (x) be a polynomial of order (2n − 1) formed by the product of a polynomial of order (n − 1) and a Legendre polynomial of order n, that is,  n−1   i ci x (Pn (x)) (E.32) R(2n−1) (x) = i=0

With this definition, the integral of R(2n−1) (x), with limits from −1 to 1, is guaranteed to be zero. To see this, we apply (E.31) to the first polynomial on the right-hand side of (E.32), integrate both sides, and then apply the orthogonality properties of Legendre polynomials (cf. (9.48)), that is,  9  1 8  1 n−1 R(2n−1) (x)dx = bi Pi (x) (Pn (x)) dx −1

−1

=

n−1 

i=0

8 bi

i=0

=

1

−1

9 Pi (x)Pn (x)dx

0

(E.33)

3. One can always decompose a (2n − 1)th order polynomial, say, ψ(2n−1) (x), into a sum of two polynomials ψ(2n−1) (x) = ζ(n−1) (x) + R(2n−1) (x)

(E.34)

where ζ(n−1) (x) is an (n − 1)th order polynomial and R(2n−1) (x) is a (2n − 1)th order polynomial that satisfies the form given in (E.32). To show this fact constructively, let r1 , . . . , rn be the roots of the n th -order Legendre polynomial, Pn (x). By virtue of the definition given in (E.32), we see that R(2n−1) (ri ) = 0 also. Using this result, we can apply each of the n roots to (E.34) and obtain ψ(2n−1) (ri ) = ζ(n−1) (ri )

i = 1, 2, . . . , n

(E.35)

th One can then obtain ζ(n−1) (x) to be  − 1) order polynomial that  the unique (n passes through n points given by ri , ψ(2n−1) (ri ) . Subsequently, R(2n−1) (x) can be found by subtracting ζ(n−1) (x) from ψ(2n−1) (x). 4. Using the decomposition given in (E.34) and the integral identity given in (E.33), an immediate consequence is the following identity:  1  1 ψ(2n−1) (x)dx = ζ(n−1) (x)dx (E.36) −1

−1

This means the integral of an (2n − 1)th order polynomial can always be replaced by the integral of a corresponding (n − 1)th order polynomial. We now use the last two results, namely (E.35) and (E.36), to determine the Gauss-Legendre parameters. Recall (E.27), which is the condition for a GaussLegendre quadrature, and apply it to ψ(2n−1) (x),  1 n  ψ(2n−1) (x)dx = Wi ψ(2n−1) (xi ) (E.37) −1

i=1

690

Appendix E: Additional Details and Fortification for Chapter 5

Now set xi = ri , that is, the roots of the n th order Legendre polynomial. Next, apply (E.35) on the right-hand side, and apply (E.36) on the left-hand side of the equation: 

1

ψ(2n−1) (x)dx

−1

1 −1

n−1 k=0

ζ(n−1) (x)dx

n−1 1 

n−1 

k

bk x dx

n 

Wi ζ(n−1) (ri )

n 

=

Wi

i=1

 bk

k=0

 n 

k=0

=

(E.38)

bk xk . Then (E.38) becomes

−1 k=0

bk

Wi ψ(2n−1) (ri )

i=1



n−1 

n  i=1



Let ζ(n−1) (x) =

=

1

−1

xk dx

Wi rik − γk

bk

=

n 

Wi rik

i=1

k=0



 bk rik

k=0

n−1 

=

 n−1 

0

(E.39)

i=1

where  γk =

1

x dx = k

−1

⎧ ⎨ 2/(k + 1)

if k is even

⎩

if k is odd

(E.40) 0

The bk value should be left arbitrary because it corresponds to a general polynomial ψ(2n−1) , as required by the second condition for a Gauss-Legendre quadrature. This then yields n equations. In matrix form, we have ⎛ ⎜ ⎜ ⎜ ⎝

1 r1 .. .

1 r2 .. .

... ... .. .

1 rn .. .

r1n−1

r2n−1

...

rnn−1

⎞⎛ ⎟⎜ ⎟⎜ ⎟⎜ ⎠⎝

W1 W2 .. . Wn

⎞

⎛

⎟ ⎜ ⎟ ⎜ ⎟=⎜ ⎠ ⎝

γ0 γ1 .. .

⎞ ⎟ ⎟ ⎟ ⎠

(E.41)

γn−1

In summary, to obtain the parameters for an n-point Gauss-Legendre quadrature, first solve for the roots ri of the n th -order Legendre polynomial, i = 1, . . . , n. After substituting these values into (E.41), we can solve for Wi , i = 1, . . . , n.4 4

 The first equation in (E.41), ni=1 Wi = 2, can be viewed as a partition of the domain −1 ≤ x ≤ 1 into n segments having widths Wi . As each of these partitions are given the corresponding heights of F (xi = ri ), the integral approximation is seen as a sum of rectangular areas. This means that the process replaces the original shape of the integration area into a set of quadrilaterals. Hence, the general term “quadrature.” For integrals of function in two dimensions, a similar process is called “cubature.”

Appendix E: Additional Details and Fortification for Chapter 5 EXAMPLE E.7.

For n = 3, we have P3 (x) =

691

 x 2 5x − 3 2

√ √ whose roots are, arranged in increasing order, r1 = − 0.6, r2 = 0 and r3 = 0.6. Substituting these values in (E.41), ⎛ ⎞ ⎛ ⎞ ⎞⎛ 1 √1 2 W1 √1 ⎝ − 0.6 0 0.6 ⎠ ⎝ W2 ⎠ = ⎝ 0 ⎠ W3 2/3 0.6 0 0.6 whose solution is given by W1 = W3 = 5/9 and W2 = 8/9. Note that r1 = −r3 . This is not a coincidence but a property of Legendre polynomials. In general, for an n th -order Legendre polynomial: (1) for n odd, one of the roots will always be zero, and (2) each positive root will have a corresponding negative root of the same magnitude.

Extending the results to p -dimensional box domains represented by mutually orthogonal coordinates: {x1 , . . . , x p }, the Gauss-Legendre formulas can be applied one dimension at a time, that is, 

1

−1

 ···

1 −1

[f (x1 , . . . , xn )] dx1 · · · dx p  =

=

1 −1

.. . n  i1 =1

 ···

···

⎡ 1 −1

⎣

n 

⎤   Wi p f x1 , . . . , x p −1 , ri p ⎦ dx1 · · · dx p −1

i p =1

n    Wi1 · · · Wi p f (ri1 , . . . , ri p )

(E.42)

i p =1

where Wi and ri are the same values obtained earlier for the one-dimensional case.

E.5 Proofs of Integral Theorems E.5.1 Proof of Green’s Lemma (Lemma 5.1) To prove the lemma, we make use of two possible descriptions of the boundary as given in (E.17) and (E.18). Recalling (E.17), the domain of the surface of integration S is given by D : ulower ≤ u ≤ uupper ;

φ0 (u) ≤ v ≤ φ1 (u)

where the closed contour C is the sum given by C = C0,v − C1,v where C0,v and C1,v , the curves described by φ0 (u) and φ1 (u), respectively, are positive with increasing values of u.

692

Appendix E: Additional Details and Fortification for Chapter 5

Applying this description to the second surface integral in (5.1),  S

∂F (u, v) du dv ∂v



uupper

= 



φ0 (u)

ulower uupper

=

φ1 (u)

 ∂F (u, v) dv du ∂v

(F (u, φ1 (u)) − F (u, φ0 (u))) du

ulower

3

=

F (u, v)du

−

(E.43)

C

Likewise, using (E.18), the domain of the surface of integration S is given by D : vlower ≤ v ≤ vupper ;

ψ0 (v) ≤ u ≤ ψ1 (v)

where the closed contour C is now equal to the sum given by C = C1,u − C0,u where C0,v and C1,v , the curves described by ψ0 (u) and ψ1 (u), respectively, are positive with increasing values of v. Applying this domain description to the first surface integral in (5.1),  S

∂G(u, v) dudv ∂u

 =



vlower

 =

vupper

ψ1 (v) ψ0 (v)

vupper

vlower

 ∂G(u, v) dv du ∂u

(G(ψ1 (v), v) − G(ψ0 (v), v)) du

3

G(u, v)dv

=

(E.44)

C

Combining (E.43) and (E.44), we arrive at the formula given in Green’s lemma, 3

3



G(u, v)dv + C

F (u, v)du = C

S



∂G(u, v) du dv − ∂u

S

∂F (u, v) du dv ∂v

E.5.2 Proof of Divergence Theorem (Theorem 5.1) In rectangular coordinates, let f be given by f = f x δx + f y δy + f z δz The volume integral in (5.5) can be expanded to be the sum of three terms 



∇ · f dV = V

V

∂ fx dV + ∂x

 V

∂ fy dV + ∂y

 V

∂ fz dV ∂z

(E.45)

Appendix E: Additional Details and Fortification for Chapter 5

Figure E.9. The normal vector to the surface x = ξmax (y, z) is given by N1 which has a magnitude equal to the differential surface, dS1 .

For the first term in (E.45), we can use the following description of the volume of integration: 5 V :

zmin ≤

z

≤ zmax

ηmin (z) ≤

y

≤ ηmax (z)

ξmin (y, z) ≤

x

≤ ξmax (y, z)

to obtain the following triple integral formulation   zmax  ηmax (z)  ξmax (y,z) ∂ fx ∂ fx dV = dx dy dz ∂x V ∂x zmax ηmin (z) ξmin (y,z) After performing the inner integration with respect to x, the result is a difference of two surface integrals  zmax  ηmax (z)  ∂ fx dV = f x (ξmax (y, z), y, z) dydz V ∂x zmin ηmin (z)  zmax  ηmax (z) − f x (ξmin (y, z), y, z) dydz (E.46) zmax

ηmin (z)

The first surface integral in (E.46) is with respect to the surface: S1: x = ξmax (y, z). To determine the differential area of the surface, dS1 at a point in the surface, we can use the position vector r of the point in surface S1 . Along the curve in the surface, in which z is fixed, we have a tangent vector given by (∂r/∂y) dy. Likewise, along the curve in the surface, in which y is fixed, we have a tangent vector given by (∂r/∂z)dz. This is shown in Figure E.9. By taking the cross product of these two tangent vectors, we obtain a vector N1 which is normal to surface S1 whose magnitude is the area of the parallelogram formed by the two tangent vectors, that is, 1 N1 = dS1 n  1 is the unit normal vector. where n Thus, with the position vector r along the surface given by r = ξmax (y, z) δx + y δy + z δz 5

This assumes that any line that is parallel to the x axis will intersect the surface boundary of region V at two points, except at the edges of the boundary, where it touches at one point. If this assumption is not true for V , it can always be divided into subsections for which this assumption can hold. After applying the divergence theorem to these smaller regions, they can be added up later, and the resulting sum can be shown to satisfy the divergence theorem.

693

694

Appendix E: Additional Details and Fortification for Chapter 5

Figure E.10. The normal vector to the surface x = ξmin (y, z) is given by N2 which has a magnitude equal to the differential surface, dS2 .

we have 

1 dS1 n

= 

= =

∂r ∂r × ∂y ∂z

 dydz

   ∂ξmax ∂ξmax δ + δy × δ + δz dydz ∂y x ∂z x   ∂ξmax ∂ξmax δx − δ − δ dydz ∂z y ∂y z

By taking the dot product of both sides with δx , n1 · δx ) dS1 = dydz (

(E.47)

The same arguments can be used for the other surface given by x = ξmin (y, z). The difference is that, as shown in Figure E.10, the normal vector N2 = (∂r/∂z) × (∂r/∂y), and thus n2 · δx ) dS2 = − dydz (

(E.48)

Returning to equation (E.46), we can now use the results in (E.47) and (E.48) to obtain,     ∂ fx  n 1 · δx ) + dV = f x ( n 2 · δx ) = f x ( f x δx · n (E.49) V ∂x S1 S2 S Following the same procedure, we could show that the other two terms in (E.45) can be evaluated to be   ∂ fy  dV = f y δy · n (E.50) V ∂y S   ∂ fz  f z δz · n dV = (E.51) V ∂z S Adding the three equations: (E.49), (E.50) and (E.51), we end up with the divergence theorem, that is,       ∂ fy ∂ fy ∂ fx  dS + + dV = f x δx + f y δy + f z δz · n (E.52) ∂x ∂y ∂z V S

Appendix E: Additional Details and Fortification for Chapter 5

Figure E.11. A small sphere of radius ρ removed from V , yielding surface S1 and S2 .

E.5.3 Proof of Green’s Theorem (Theorem 5.2) First, we have ∇ · (φ∇ψ)

=

(∇φ · ∇ψ) + φ∇ 2 ψ

∇ · (ψ∇φ)

=

(∇ψ · ∇φ) + ψ∇ 2 φ

Subtracting both equations, ∇ · (φ∇ψ − ψ∇φ) = φ∇ 2 ψ − ψ∇ 2 φ Then taking the volume integral of both sides, and applying the divergence theorem,    2   dS = φ∇ ψ − ψ∇ 2 φ dV (φ∇ψ − ψ∇φ) · n S

V

E.5.4 Proof of Gauss’ Theorem (Theorem 5.3) Suppose the origin is not in the region bounded by S. Then,   1 1 1 ∇ · 2 δr = ∇ 2 · δr + 2 ∇ · δr r r r   2 1 2 = − 3 δr · δr + 2 r r r =

0

Thus with the divergence theorem,       1 1  dS = δr · n ∇ · 2 δr dV = 0 2 r r S V Next, suppose the origin is inside S. We remove a small sphere of radius ρ , which leaves a region having two surfaces: the original surface S1 and a spherical surface inside given by S2 (see Figure E.11). & & is now bounded by S1 and S2 . Because the region in V The reduced volume V satisfies the condition that the origin is not inside, we conclude that    1 1 1   dS = 0 ∇ · 2 δr dV = δ · n dS + δ ·n 2 r 2 r r r r & V S1 S2 Focusing on S2 , the unit normal is given by −δr , and  1 1 1   dS = 4π δ ·n=− 2 → δ ·n 2 r r2 r r r S2

695

696

Appendix E: Additional Details and Fortification for Chapter 5

Thus if the origin is inside S = S1 ,  S

1  dS = 4π δ ·n r2 r

E.5.5 Proof of Stokes’ Theorem (Theorem 5.4) Let S be parameterized by u and v, then, 3 3 f · dr = f x dx + f y dy + f zdz C

3

C

=



 3   ∂x ∂x ∂y ∂y du + dv + fy du + dv ∂u ∂v ∂u ∂v C   3 ∂z ∂z + fz du + dv ∂u ∂v C

fx C

3 =

f (u, v)du + g(u, v)dv

(E.53)

C

where, ∂x ∂y ∂z + fy + fz ∂u ∂u ∂u ∂x ∂y ∂z g(u, v) = f x + fy + fz ∂v ∂v ∂v Applying Green’s lemma, (5.1), to (E.53)    3 ∂g ∂f − du dv ( f (u, v)du + g(u, v)dv) = ∂v C S ∂u f (u, v)

=

fx

(E.54)

The integrand of the surface integral in (E.54) can be put in terms of the curl of f as follows:   ∂ f y ∂y ∂g ∂ f x ∂x ∂f ∂2x ∂2y ∂ f z ∂z ∂2z − = + fx + + fy + + fz ∂u ∂v ∂u ∂v ∂v∂u ∂u ∂v ∂v∂u ∂u ∂v ∂v∂u   2 2 ∂ f y ∂y ∂ f x ∂x ∂ x ∂ y ∂ f z ∂z ∂2z − + fx + + fy + + fz ∂v ∂u ∂u∂v ∂v ∂u ∂u∂v ∂v ∂u ∂u∂v   ∂F k ∂m ∂k   ∂F k ∂m ∂k = − ∂m ∂u ∂v m=x,y,z ∂m ∂v ∂u m=x,y,z 

=

=

=

k=x,y,z



k=x,y,z



 ∂ f y ∂(x, y) ∂ f y ∂(z, y) ∂ f x ∂(y, x) ∂ f x ∂(z, x) + + + ∂y ∂(u, v) ∂z ∂(u, v) ∂x ∂(u, v) ∂z ∂(u, v)   ∂ f z ∂(x, z) ∂ f z ∂(y, z) + + ∂x ∂(u, v) ∂y ∂(u, v)     ∂ fy ∂ f x ∂(x, y) ∂ f z ∂ f y ∂(y, z) − + − ∂x ∂y ∂(u, v) ∂y ∂z ∂(u, v)   ∂ f z ∂(z, x) ∂ fx − + ∂z ∂x ∂(u, v)   ∂(y, z) ∂(z, x) ∂(x, y) δ + δ + δ (E.55) (∇ × f ) · ∂(u, v) x ∂(u, v) y ∂(u, v) z

Appendix E: Additional Details and Fortification for Chapter 5

697

 dS is given by Recall that n 

 dS = n

 ∂(y, z) ∂(z, x) ∂(x, y) δx + δy + δz du dv ∂(u, v) ∂(u, v) ∂(u, v)

(E.56)

Combining (E.53), (E.54), (E.55) and (E.56), will yield 3

  dS (∇ × f ) · n

f · dr = C

S

which is Stokes’ theorem.

E.5.6 Proof of Leibnitz formulas 1. One-Dimensional Case (Theorem 5.6). Using the definition of a derivative: d dα





h(α)

F (α, x) dx

=

g (α)

1 lim

α → 0 α



h(α+ α)

F (α + α, x)dx

g(α+ α)

 −



h(α)

F (α, x)dx g(α)

(E.57) The first integral in the left-hand side of (E.57) can be divided into three parts, 

h(α+ α)

g (α+ α)



h(α+ α)

F (α + α, x) dx = F (α + α, x) dx h(α)  h(α)  g (α) + F (α + α, x) dx + F (α + α, x) dx g (α)

g (α+ α)

(E.58)

Furthermore, the first integral in the left-hand side of (E.58) can be approximated by the trapezoidal rule, 

h(α+ α) h(α)

 1  F α + α, h(α+ α) 2     + F α + α, h(α) h(α+ α) − h(α)

F (α + α, x) dx ≈

(E.59)

Likewise, we can also approximate the third integral in the left-hand side of (E.58) as 

g (α)

g (α+ α)

 1  F α + α, g (α+ α) 2     + F α + α, g (α) g (α) − g (α+ α)

F (α + α, x) dx ≈

(E.60)

698

Appendix E: Additional Details and Fortification for Chapter 5

Substituting (E.59) and (E.60) into (E.58), and then into (E.57), d dα



8

g (α)

 =



 F (α + α, x) − F (α, x) F (α, x) dx = lim dx

α→0

α g(α))      F α + α, h(α+ α) + F α + α, h(α)  + h(α+ α) − h(α) 2 α 9      F α + α, g (α+ α) + F α + α, g (α)  + g (α) − g (α+ α) 2 α

h(α)

h(α)

g(α))

h(α)

∂ dh dg F (α, x) dx + F (α, h(α)) − F (α, g(α)) ∂α dα dα

2. Three-Dimensional Case (Theorem 5.7). From the definition of the derivative,  d f (x, y, z, α)dV dα V (α) ' (   1 = lim f (x, y, z, α + α)dV − f (x, y, z, α)dV

α→0 α V (α+ α) V (α)  By adding and subtracting the term V (α) f (x, y, z, α + α)dV in the right-hand side, d dα

 f (x, y, z, α)dV V (α)

1 = lim

α→0 α

'

(

 f (x, y, z, α + α)dV −

V (α)

f (x, y, z, α)dV V (α)

' (  1 + lim f (x, y, z, α + α)dV − f (x, y, z, α + α)dV

α→0 α V (α+ α) V (α) '  ∂f 1 dV + lim = f (x, y, z, α + α)dV

α→0 α V (α) ∂α V (α+ α) (  (E.61) f (x, y, z, α + α)dV − V (α)

The last group of terms in the right-hand side (E.61) is the difference of two volume integrals involving the same integrand. We can combine these integrals by changing the volume of integration to be the region between V (α + α) and V (α).   f (x, y, z, α + α)dV − f (x, y, z, α + α)dV = V (α+ α)



V (α)

f (x, y, z, α + α)dV

(E.62)

V (α+ α)−V (α)

We could approximate the differential volume in (E.62) as the parallelepiped formed by the vectors (∂r/∂u)du, (∂r/∂v)dv and (∂r/∂α)dα, where u and v are parameters used to describe surface S(α). This is shown in Figure E.12.

Appendix E: Additional Details and Fortification for Chapter 5

699

Figure E.12. Graphical representation of differential volume emanating from points in S(α) towards S(α + α).

Recall that



   ∂r ∂r  dS du × dv = n ∂u ∂v

which then gives a differential volume attached to S(α)   ∂r ∂r ∂r dV |(x,y,z)∈V (α+ α)−V (α) = · × dα du dv ∂α ∂u ∂u ∂r  dα dS ·n ∂α The volume integral for points bounded between the surfaces of V (α) and V (α + α) can now be approximated as follows:   ∂r  α dS f (x, y, z, α + α)dV ≈ f (x, y, z, α + α) ·n ∂α V(α+ α) −V(α) S(α) =

(E.63) Substituting (E.63) into (E.62) and then to (E.61),   d ∂f f (x, y, z, α)dV = dV dα V (α) ∂α V (α)  1 ∂r  αdS + lim f (x, y, z, α + α) ·n

α→0 α S(α) ∂α  = V (α)

∂f dV + ∂α

 f (x, y, z, α) S(α)

∂r  dS ·n ∂α

which is the Leibnitz rule for differentiation of volume integrals.

APPENDIX F

Additional Details and Fortification for Chapter 6

F.1 Supplemental Methods for Solving First-Order ODEs F.1.1 General Ricatti Equation In some cases, the solution of a first-order differential equation is aided by increasing the order to a second-order differential equation. One such case is the generalized Ricatti differential equation given by the following general form: dy = P(x)y2 + Q(x)y + R(x) dx

(F.1)

Note that when P(x) = 0, we have a first-order linear differential equation, and when R(x) = 0, we have the Bernouli differential equation. Using a method known as the Ricatti transformation, y(x) = −

1 dw P(x)w dx

we obtain dy dx

=

Py2

=

Qy

=

1 d2 w 1 − + 2 Pw dx Pw2   1 dw 2 Pw2 dx −



dw dx

2

1 dP + 2 p w dx



dw dx



Q dw Pw dx

which then reduces (F.1) to be   d2 w dP(x)/dx dw − + Q(x) + P(x)R(x)w = 0 dx2 P(x) dx

(F.2)

Note that (F.2) is a second-order ordinary differential equation. Nonetheless, it is a linear differential equation, which is often easier to solve than the original nonlinear first-order equation.

700

Appendix F: Additional Details and Fortification for Chapter 6 EXAMPLE F.1.

701

Consider the following differential equation: dy 2 1 = xy2 − y − 3 dt x x

Noting that P(x) = x, Q(x) = −2/x and R(x) = −1/x3 , the Ricatti transformation y = −(dw/dx)/(xw) converts it to a linear second-order differential equation given by d2 w dw +x −w=0 dx2 dx which is a Euler-Cauchy equation (cf. Section 6.4.3). Thus we need another transformation z = ln(x), which would transform the differential equation further to be x2

d2 w =w dz2 whose solution becomes, w(z) = Ae−z + Bez

→

1 w(x) = A + Bx x

Putting it back in terms of y, 1 dw y=− = xw dx

A −B 2 1 C − x2 x = 2 A x C + x2 x + Bx x

where C = A/B is an arbitrary constant.

F.1.2 Legendre Transformations Usually, methods that introduce a change of variables involve only transformations from the original independent and dependent variables, say, x and y, to new independent and dependent variables, say, p and q. In some cases, however, we need to consider the derivatives as separate variables in the transformations, for example,   dy p = p x, y, dx   dy q = q x, y, dx   dq dq dy = x, y, dp dp dx (F.3) These types of transformations are called contact transformations. One particular type of contact transformation is the Legendre transformation. This type of transformation is very useful in the field of thermodynamics for obtaining equations in which the roles of intensive and extensive variables need to be switched in a way that conserves the information content of the original fundamental equations. In the case here, the Legendre transformation is used to solve differential equations.

702

Appendix F: Additional Details and Fortification for Chapter 6 y

(x,y)

p

Figure F.1. Description of a curve as an envelope of tangent lines used for Legendre transformation rules.

x -q

The Legendre transformation takes a curve y = y(x) and obtains an equivalent description by using an envelope generated by a family of tangent lines to the curve at the point (x, y), that is, y = p x + (−q)

(F.4)

where p is the slope and −q is the y-intercept. This is illustrated in Figure F.1. The Legendre transformation uses the following transformations: p=

dy ; dx

q=x

dy − y and dx

dq =x dp

(F.5)

where p is the new independent variable and q is the new dependent variable. The inverse Legendre transformations are given by x=

dq ; dp

y=p

dq −q dp

and

Now consider the differential equation   dy f x, y, =0 dx In terms of the new variables, we have   dq dq f ,p − q, p dp dp

dy =p dx

(F.6)

(F.7)

(F.8)

It is hoped that (F.8) will be easier to solve than (F.7), such as when the derivative dy/dx appears in nonlinear form while x and y are in linear or affine forms. If this is the case, one should be able to solve (F.8) to yield a solution of the form given by: S(p, q) = 0. To return to the original variables, we observe that   ∂S dq ∂S + = 0 → g(p, xp − y) + h(p, xp − y)x = 0 ∂p ∂q dp where g and h are functions resulting from the partial derivatives. Together with f (x, y, p ) = 0, one needs to remove the presence of p to obtain a general solution s(x, y) = 0. In some cases, if this is not possible, p would have to be left as a parameter, and the solution will be given by curves described by x = x(p ) and y = y(p ).

Appendix F: Additional Details and Fortification for Chapter 6

703

In particular, Legendre transformations can be applied to a differential equations given by y = xψ(p ) + η(p )

(F.9)

where ψ(p ) = p .1 For instance, one may have a situation in which the dependent variable y is modeled empirically as a function of p = dy/dx in the form given by (F.9). After using the Legendre transformation, we arrive at     dq 1 η(p ) + q= dp ψ(p ) − p p − ψ(p ) which is a linear differential equation in variables p and q. EXAMPLE F.2.

Consider the differential equation given by  2 dy dy =x +y dx dx

then after the Legendre transformation, we obtain dq 1 p − q= dp 2p 2 whose solution is given by q=

p2 √ +C p 3

After taking the derivative dq/dp , dq 2 C√ =x= p+ p dp 3 2 Unfortunately, p (x) is not easily found. Instead, we could treat p as a parameter, that is, x = x(p ), and insert this back to the given equation to obtain 2 C√ α+ α 3 2 where α is a parameter for the solution (y(α), x(α)), and C is an arbitrary constant. y = −x(α) α + α2 ; subject to x(α) =

F.2 Singular Solutions For some differential equations, a solution may exist that does not have arbitrary constants of integration. These solutions are called singular solutions. Singular solutions, if they exist for a differential equation, have a special property that it is the envelope of the general solutions. Thus their utility is often in the determination of the bounds of the solution domain. For a first-order differential equation,   dy =0 (F.10) f x, y, dx the general solution is given by φ(x, y, C) = 0 1

If ψ(p ) = p , an algebraic equation results, that is, q = −η(p ).

(F.11)

704

Appendix F: Additional Details and Fortification for Chapter 6

where C is an arbitrary constant. For φ to be a singular solution, it should not be a function of the arbitrary constant C. Thus ∂φ = S(x, y) = 0 ∂C

(F.12)

where S(x, y) is obtained with the aid of (F.11). To determine whether S(x, y) = 0 is indeed a singular solution, one needs to check if ∂S ∂S dy + =0 ∂x ∂y dx

(F.13)

will satisfy the original differential equation (F.10). If it does, then it is a singular solution. EXAMPLE F.3.

Clairaut’s equation is given by  2 dy dy y=x + dx dx

(F.14)

Using the quadratic equation to find dy/dx as an explicit function of x and y, this can be rearranged to give 7   dy x y = −1 ± 1 + 4 2 dx 2 x which is an isobaric equation (cf. (6.13)). By introducing a new variable, u = y/x2 , the original differential equation can be reduced to a separable equation, that is, du 1 dx = − √ 2 x 4u + 1 ± 1 + 4u whose solution is given by √ ln( 4u + 1 ± 1) = ln(x) + k

→

y = Cx + C2

where C is an arbitrary constant. To search for the singular function, following (F.11) yields φ(x, y, C) = y − Cx − C2 = 0

(F.15)

then ∂φ = −x − 2C = 0 (F.16) ∂C where C can be eliminated from (F.16) using (F.15) to obtain " x2 S(x, y) = ± x2 + 4y = 0 → y = − (F.17) 4 Finally, we can check that (F.17) satisfies (F.14), thus showing that (F.17) is indeed a singular solution of (F.14). A simpler alternative approach to solving Clairaut’s equation is to take the derivative of (F.14) with respect to x while letting p = dy/dx, that is, dp dp + 2p dx dx

p

=

p +x

0

=

dp (x + 2p ) dx

Appendix F: Additional Details and Fortification for Chapter 6

705

30

C = −5

C=1

Figure F.2. Plot of the general solution y1 (x) = Cx + C2 (dotted lines) and the singular solution y2 (x) = −x2 /4 (solid curve).

y1(x;C), y2(x)

20

10

C=3

C = −1

0

−10

−20

−30 −10

−5

0

5

x

then dp =0 and dx yielding two solutions of different forms y1 = c1 x + c2

and

p =−

y2 = −

x 2

x2 + c3 4

Substituting both solutions into (F.14) will result in c3 = 0 and c2 = c21 . Thus the general solution is given by y1 = cx + c2 while the singular solution is given by y2 = −

x2 4

If we plot the general solution y1 (x) = cx + c2 and the singular solution, y2 = −x2 /4, we see in Figure F.2 that the singular solution is an envelope for the general solution.

F.3 Finite Series Solution of dx/dt = Ax + b(t) The method shown here solves the linear equation with constant coefficient A given by d x = Ax + b(t) dt subject to x(0) = x0 . It is applicable also to matrices A that are not diagonalizable. The steps of the procedure are given as follows: 1. Let the vector of eigenvalues of A[=]n × n be grouped into p distinct sets of repeated eigenvalues, that is,      λ = λ1 · · · λ p with λi = λi · · · λi [=] 1 × mi p where λi = λk when i = k, and i=1 mi = n.

10

706

Appendix F: Additional Details and Fortification for Chapter 6

2. Next, define the matrix Q[=]n × n, ⎛ ⎞ ⎛ Q1 q0,0 (λi ) ⎜ .. ⎟ ⎜ .. Q = ⎝ . ⎠ Qi = ⎝ . qmi −1,0 (λi )

Qp where,

qj, (λ) =

⎧ ⎪ ⎪ ⎪ ⎨

⎞ q0,n−1 (λi ) ⎟ .. ⎠ [=] mi × n (F.18) . qmi −1,n−1 (λi )

··· .. . ···

0

if

> [t,y]=ode45(@derfunc,[0,2],[2;1]); where [t,y] are the output time and states, respectively, derfunc is the file name of the derivative function, [0,2] is the time span, and [2;1] is the vector of initial values. A partial list of solvers that are possible alternatives to ode45 is given in Table G.1. It is often suggested to first try ode45. If the program takes too long, then it could be due to the system being stiff. In those cases, one can attempt to use ode15s. There are more advanced options available for these solvers in MATLAB. In addition to the ability to set relative errors or absolute errors, one can also include “event handling” (e.g., modeling a bouncing ball), allow passing of model parameters, or solving equations in mass-matrix formulations, that is, M(t, y)

d y = f (t, y) dt 715

716

Appendix G: Additional Details and Fortification for Chapter 7 Table G.1. Some initial value solvers for MATLAB Solver

Description

Remarks

ode23

(2, 3)th Bogacki-Shampine Embedded Runge-Kutta

ode45

(4, 5)th Dormand-Prince Embedded Runge-Kutta

ode113

Adams-Bashforth-Moulton Predictor-Corrector

ode15s

Variable-order BDF (Gear’s method)

ode23s

Order-2 Rosenbrock method

ode23t

Trapezoid method

ode23tb

Trapezoid method stage followed by BDF stage

Efficient for most non-stiff problems. Also for non-stiff problems. Used when state-space model is more computationally intensive. For stiff problems. May not be as accurate as ode45. Allows settings/definition of Jacobians. Can be used to solve DAE problems with index-1. For stiff problems. May solve problems where ode15s fails. For stiff problems. Implements some numerical damping. Used also to solve DAE problems with index-1. For stiff problems. May be more efficient than ode15s at crude tolerances.

where M(t, y) is either singular (as it would be for DAE problems) and/or has preferable sparsity patterns.

G.1.2 BVP Solver As a quick example, consider the same system as (G.1), but instead of the initial conditions, we wish to satisfy the following two-point boundary conditions: y1 (1) = 0.3 and y2 (0) = 1. Then the following steps are needed to solve this boundary value problem in MATLAB: 1. Build the model file, say, derfunc.m, as done in the previous section. 2. Let r be the vector of residuals from the boundary conditions; that is, reformulate the boundary conditions in a form where the the right hand side is made equal to zero,  r=

y1 (1) − 0.3 y2 (0) − 1



Now build another file, say, bconds.m, that generates r, function r1 = r2 = r =

r = bconds(yinit,yfinal) yfinal(1)-0.3 ; yinit(2)-1 ; [r1;r2] ;

Appendix G: Additional Details and Fortification for Chapter 7

Note that this file does not know that the final point is at t = 1. That information will have to come from a structured data, trialSoln, that is formed in the next step. 3. Create a trial solution data, trialSoln, >> trialSoln.x = linspace(0,1,10); >> trialSoln.y = [0.5;0.2]*ones(1,10); The data in trialSoln.x give the initial point t = 0, final point t = 1, and 10 mesh points. One can vary the mesh points so that finer mesh sizes can be focused around certain regions. The data in trialSoln.y just give the intial conditions repeated at each mesh point. This could also be altered to be closer to the final solution. (Another MATLAB command bvpinit is available to create the same initial data and has other advanced options.) 4. Run the BVP solver, >> soln = bvp4c(@derfunc,@bconds,trialSoln); The output, soln, is also a structured data. Thus for plotting or other postprocessing of the output data, one may need to extract the t variable and y variables as follows: >> t=soln.x;

y=soln.y;

There are several advanced options for bvp4c, including the solution of multipoint BVPs, some singular BVPs, and BVPs containing unknown parameters. The solver used in bvp4c is said to be finite difference method coupled with a three-stage implicit Runge-Kutta method known as Lobatto III-a.

G.1.3 DAE Solver Consider the van der Pol equation in Lienard coordinates given by dy1 = −y2 dt   y32 0 = y1 − − y2 3 which could be put into the mass matrix form as       −y2 d 0 1 y1   3 = y 0 0 y2 y1 − 32 − y2 dt The following steps are needed to solve this DAE problem using MATLAB. 1. Build the model file, say, daevdpol.m, function dy = daevdpol( t, y ) y1 = y(1) ; y2 = y(2) ; dy1 = -y2 ; dy2 = y1 - (y2ˆ3/3 - y2) ; dy = [dy1;dy2] ; 2. Make sure the initial conditions are consistent. For instance, the algebraic condition is satisfied for y = (−0.0997, 0.1)T .

717

718

Appendix G: Additional Details and Fortification for Chapter 7

3. Set the parameter, options, to include the mass matrix information using the command >> options=odeset(’Mass’,[1,0;0,0]); 4. Run the DAE solver >>[t,y]=ode15s(@daevdpol,[0,2],[-0.0997;0.1],options) where [0,2] is the time span.

G.2 Derivation of Fourth-Order Runge Kutta Method G.2.1 Fourth-Order Explicit RK Method To obtain a fourth-order approximation, we truncate the Taylor series expansion as follows:     dy  h2 d2 y  h3 d3 y  h4 d4 y  yk+1 ≈ yk + h + + + (G.2) dx  2! dx2  3! dx3  4! dx4  tk ,yk

tk ,yk

tk ,yk

tk ,yk

i

The coefficient of h in (G.2) can then be matched with the coefficients of h in (7.13). This approach is very long and complicated. For instance, by expanding the derivatives of y in terms of f and its partial derivatives, dy dt

=

f

d2 y dt2

=

∂f ∂f + f ∂t ∂y

d3 y dt3

=

∂2 f ∂2 f ∂ 2 f 2 ∂f ∂f + 2f + f + + ∂t2 ∂t∂y ∂y2 ∂t ∂y



∂f ∂y

2 f

.. . The number of terms increases exponentially with increases in the order of differentiation. These equations, including those for higher orders, can be made more tractable using an elegant formulation using labeled trees (see, e.g., Hairer and Wanner [1993]). As an alternative, we simplify the process by picking specific forms for f (t, y). The first choice is to let f (t, y) = t3 . The analytical solution from yk to yk+1 is given by d y dt

=

t3

(tk + h)4 − tk4 4 3 1 yk+1 = yk + tk3 h + tk2 h2 + tk h3 + h4 2 4 Applying a four-stage Runge-Kutta method using (7.12) and (7.13), yk+1 − yk

=

δk1

=

h tk3

δk2

=

h (tk + a2 h)3

δk3

=

h (tk + a3 h)3

(G.3)

Appendix G: Additional Details and Fortification for Chapter 7

δk4

=

h (tk + a4 h)3

yk+1

=

yk + c1 δk1 + c2 δk2 + c3 δk3 + c4 δk4

=

yk + (c1 + c2 + c3 + c4 ) tk3 h + 3 (c2 a2 + c3 a3 + c4 a4 ) tk2 h2   + 3 c2 a22 + c3 a23 + c4 a24 tk h3   + c2 a32 + c3 a33 + c4 a34 tk h4

719

(G.4)

Comparing (G.3) and (G.4), c1 + c2 + c3 + c4

=

1

c2 a2 + c3 a3 + c4 a4

=

c2 a22 + c3 a23 + c4 a24

=

c2 a32 + c3 a33 + c4 a34

=

1 2 1 3 1 4

(G.5)

Next, we choose f (t, y) = ty. The analytical solution is given by

 ln

d y dt 

yk+1 yk

yk+1

= = =

ty (tk + h)2 − tk2 2   (tk + h)2 − tk2 yk exp 2

The Taylor series expansion is given by, '  1 yk+1 = yk 1 + tk h + 1 + tk2 h2 2   1 1 tk + tk3 h3 2 6   ( 1 1 2 1 4 4 5 + t + t h + O(h ) 8 4 k 24 k

(G.6)

(G.7)

Applying the four-stage Runge-Kutta method using (7.12) and (7.13), δk1 δkj

= =

h tk yk

⎛

h (tk + a j h) ⎝yk +

j −1 

⎞ bj δk ⎠

j = 2, 3, 4

=1

yk+1

= =

yk + c1 δk1 + c2 δk2 + c3 δk3 + c4 δk4    yk 1 + σ1,1 tk h + σ2,0 + σ2,2 tk2 h2   σ3,1 tk + σ3,3 tk3 h3    σ4,0 + σ4,2 tk2 + σ4,4 tk4 h4 + O(h5 )

(G.8)

720

Appendix G: Additional Details and Fortification for Chapter 7

where,

σ1,1

=

4 

ci

i=1

σ2,0

=

4 

ci ai

i=2

σ2,2

σ3,1

=

=

4 

ci

i=2

j =1

4 

i−1 

ci

=

4  i=3

σ4,0

=

4 

ci

=

4 

=

4 

ci ai

i−1 

bij

j =1

i=2

bij bj

=1 j =+1

ci ai

i−1 

bij a j

j =2

ci ai

i−1 i−2   =1 j =+1

i=3

σ4,4

a bi, +

i−1 i−2  

i=3

σ4,2

bij

=2

i=3

σ3,3

i−1 

bij bj +

4 

ci

i=3

i−1  =2

bi a

−1 

bj + c4 b43 b32 a2

j =1

c4 b43 b32 b21

Now compare the coefficients of (G.8) and (G.7). Using (7.17) and including (G.5), we end up with the eight equations necessary for the fourth-order approximation:

c1 + c2 + c3 + c4

=

1

c3 b32 a2 + c4 (b43 a3 + b42 a2 )

=

1 6

c2 a2 + c3 a3 + c4 a4

=

1 2

c3 a3 b32 a2 + c4 a4 (b43 a3 + b42 a2 )

=

1 8

c2 a22 + c3 a23 + c4 a24

=

1 3

  + c4 b43 a23 + b42 a22

=

1 12

c2 a32 + c3 a33 + c4 a34

=

1 4

c4 b43 b32 a2

=

1 24

c3 b32 a22

(G.9)

 After replacing a j with  bj , there are ten unknowns (bij and c j , i < j , j = 1, 2, 3, 4) with only eight equations, yielding two degrees of freedom. One choice is to set b31 = b41 = 0. This will result in the coefficients given in the tableau shown in (7.14).

Appendix G: Additional Details and Fortification for Chapter 7

Another set of coefficients that satisfies the eight conditions given in (G.9) is the Runge-Kutta tableau given by ⎛

⎞

⎜ ⎜ ⎜ ⎜ a ⎜ ⎜ ⎜ ⎜ ⎜ ⎝

⎟ ⎟ ⎟ ⎟ ⎟ ⎟= ⎟ ⎟ ⎟ ⎠

B

0

0

0

0

0

1 3

1 3

0

0

0

2 3

− 31

1

0

0

1

1

−1

1

0

1 8

3 8

3 8

1 8

cT

(G.10)

G.2.2 Fourth-Order Implicit Runge Kutta (Gauss-Legendre) Let us now obtain the two-stage implicit Runge Kutta method that yields a fourthorder approximation.1 We begin by choosing f (t, y) = y from which the full implicit formulation becomes

or



δk1 δk2

δk1

=

h (yk + b11 δk1 + b12 δk2 )

δk2

=

h (yk + b21 δk1 + b22 δk2 )



 =

(1/h) − b11 −b21

−b12 (1/h) − b22

−1 

1 1

 yk

Substituting into (7.13), yk+1

= =







(1/h) − b11 −b21   1 + p 1 h + p 2 h2 yk 1 + q1 h + q2 h2 yk +

c1

c2

−b12 (1/h) − b22

−1 

1 1

 yk (G.11)

where p1

=

c1 + c2 − b11 − b22

p2

=

c1 (b12 − b22 ) + c2 (b21 − b11 ) + b22 b11 − b12 b21

q1

=

−b11 − b22

q2

=

b11 b22 − b12 b21

The analytical solution of dy/dt = y is yk+1 = yk eh . In light of the rational form given in (G.11), we can use a fourth-order Pade’ approximation of eh instead of the Taylor series expansion, that is,   1 + (h/2) + (h2 /12) (G.12) yk+1 = yk 1 − (h/2) + (h2 /12) 1

The usual development of the Gauss-Legendre method is through the use of collocation theory, in which a set of interpolating Lagrange polynomials is used to approximate the differential equation at the collocation points. Then the roots of the s-degree Legendre polynomials are used to provide the collocation points. See, e.g., Hairer, Norsett and Wanner (1993).

721

722

Appendix G: Additional Details and Fortification for Chapter 7

Matching the coefficients of (G.11) and (G.12), we obtain 1 2 1 12 1 − 2 1 12

=

c1 + c2 − b11 − b22

=

c1 (b12 − b22 ) + c2 (b21 − b11 ) + b22 b11 − b12 b21

=

−b11 − b22

=

b11 b22 − b12 b21

(G.13)

which still leaves two degrees of freedom. A standard choice is to use the roots of the second-degree Legendre polynomial to fix the values of a1 and a2 ,2 that is, P2 (t) = t2 − t + (1/6) yielding the roots

√ 1 3 a1 = − 2 6

√ 1 3 a2 = + 2 6

and

Also, recall the consistency condition (7.17), √ 1 3 − = b11 + b12 and 2 6

√ 1 3 + = b21 + b22 2 6

(G.14)

(G.15)

From (G.13) and (G.15), √ √ we find that: c1 = c2 = 1/2, b11 = b22 = 1/4, b12 = 1/4 − 3/6 and b21 = 1/4 + 3/6.

G.3 Adams-Bashforth Parameters To determine the values of bj for the Adams-Bashforth method, we choose f (y) that facilitates the determination of the coefficients. The simplest choice is f (y) = y. Doing so, the n th - order Adams-Bashforth method becomes yk+1 = yk + h

m 

m    bj f yk−j = yk + h bj yk−j

j =0

(G.16)

j =0

where m = n − 1. With f (y) = y, the analytical solution of yk+ starting at yk is given by yk+ = eh yk

(G.17)

Substituting this relationship to (G.16) results in eh = 1 + h

m 

bj e−jh

j =0

which when expanded using Taylor’s series will yield   m  h2 h3 (jh)2 (jh)3 1+h+ + + ··· = 1 + h − bj 1 − jh + + ··· 2! 3! 2! 3! j =0

2

See Section 9.2 for a discussion on Legendre polynomials.

(G.18)

Appendix G: Additional Details and Fortification for Chapter 7

h h2 1+ + + ··· 2! 3!

=

m  j =0

⎛ =

⎝

  (jh)2 (jh)3 bj 1 − jh + − + ··· 2! 3!

m 

⎞

⎛

bj ⎠ − h ⎝

j =0

m  j =0

⎞

⎞ ⎛ m 2  h ⎝ j bj ⎠ + j 2 bj ⎠ + · · · 2! j =0

By comparing the different coefficients of h on both sides we get + m  (−1) if  > 0 j =1 j bj = m b if  = 0 +1 j j =0 or in matrix form, ⎛ ⎜ 1 ⎜ ⎜ ⎜ ⎜ 0 ⎜ ⎜ ⎜ . ⎜ .. ⎜ ⎜ ⎝ 0

1

1

···

1

2

···

.. .

.. .

..

1

m

···

2

.

⎞ ⎛ b ⎟⎜ 0 ⎟ ⎜ ⎟⎜ ⎟ ⎜ ⎟⎜ ⎟ ⎜ ⎟⎜ ⎟ ⎜ m ⎟ ⎜ b1 ⎟ ⎜ ⎟⎜ ⎟ ⎜ ⎟⎜ ⎟=⎜ ⎜ ⎜ ⎟ .. ⎟ ⎜ .. ⎟ ⎜ . ⎟⎜ . ⎟ ⎟ ⎜ ⎟⎜ ⎟ ⎜ mm ⎠ ⎝ bm ⎠ ⎝ ⎞⎛

⎞

1

1

1 2 .. . (−1)m m+1 −

⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠

(G.19)

G.4 Variable Step Sizes for BDF For variable step sizes, the coefficients of the multistep methods will no longer be constant. In this section, we treat only the BDF formulas. The approach should generally be similar for the other multistep methods. Let hk be the step size at tk and put the BDF equation (7.38) into an equivalent form,3 m 

  γ(i|k) yk−i = hk f yk+1

(G.20)

i=−1

Using the same technique of finding the necessary conditions by the simple application of the approximation to dy/dt = y, that is, f (y) = y and y = et y0 , we note that yk−j = e(tk−j −tk+1 ) yk+1 Then (G.20) reduces to m 

m  i=−1

3

 γ(i|k)

γ(i|k) e(tk−i −tk+1 ) yk+1

i=−1

(tk−i − tk+1 )2 1 + (tk−i − tk+1 ) + + ··· 2!

=

hk yk+1

=

hk



The form (G.20), in which the derivative function f is kept on one side without unknown coefficients, is often preferred when solving differential algebraic equations (DAE).

723

724

Appendix G: Additional Details and Fortification for Chapter 7

For the p th -order approximation, we again let m = p − 1, and the equation will yield ⎛ 1

⎜ ⎜ ⎜0 ⎜ ⎜ ⎜ ⎜0 ⎜ ⎜. ⎜. ⎜. ⎜ ⎝ 0

1

1

...

(tk+1 − tk )

(tk+1 − tk−1 )

...

(tk+1 − tk )2

(tk+1 − tk−1 )2

...

.. .

.. .

..

(tk+1 − tk ) p

(tk+1 − tk−1 ) p

...

.

⎞⎛ 1

γ(−1|k)

⎞

⎛

⎞ 0

⎟ ⎟⎜ ⎟ ⎜ ⎟ ⎟⎜ ⎟ ⎜ ⎜ ⎜ ⎟ ⎟ (tk+1 − tk−p +1 ) ⎟ ⎜ γ(0|k) ⎟ ⎜ − hk ⎟ ⎟ ⎟ ⎟⎜ ⎟ ⎜ ⎟ ⎟ ⎜ 2⎟⎜ (tk+1 − tk−p +1 ) ⎟ ⎜ γ(1|k) ⎟ = ⎜ 0 ⎟ ⎟ ⎟⎜ ⎟ ⎜ ⎟⎜ . ⎟ ⎜ . ⎟ .. ⎟⎜ . ⎟ ⎜ . ⎟ . ⎟⎜ . ⎟ ⎜ . ⎟ ⎟ ⎟⎜ ⎟ ⎜ ⎠ ⎠⎝ ⎠ ⎝ p 0 γ(p −1|k) (tk+1 − tk−p +1 ) (G.21)

Because the right-hand side is just −hk e2 , this equation can be solved directly using Cramer’s rule and using the determinant formulas of Vandermonde matrices. The results are ⎧ p  ⎪ tk+1 − tk ⎪ ⎪ if  = −1 ⎪ ⎪ ⎪ t k+1 − tk−j ⎪ j =0 ⎨ γ(|k) = (G.22)      p ⎪ ⎪ t − t − t t k+1 k−j k+1 k ⎪ ⎪ − if  ≥ 0 ⎪ ⎪ ⎪ tk+1 − tk− tk− − tk−j ⎩ j ≥0,j =

Note that this formula involves product terms that are the Lagrange formulas used in polynomial interpolation, which is how most textbooks derive the formulas for BDF coefficients. The approach taken here, however, has the advantage that it automatically fixes the order of approximation when we truncated the Taylor series of the exponential functions based on the chosen order. When the step sizes are constant, that is, hk = h, then (tk+1 − tk−j ) = (j + 1)h, and (G.22) can be used to find γ independent of k. For instance, for the sixth-order BDF method, that is, p = 6, the coefficient of yk−3 becomes 1 γ3 = − 4



1 2 3 5 6 · · · · 1−4 2−4 3−4 5−4 6−4

 =

15 4

To determine the appropriate value for hk , we can first set hk = hk−1 and then use either of the error-control methods given in Section G.5 to modify it. The stepdoubling approach might be simpler for the general nonlinear case.

G.5 Error Control by Varying Step Size To improve accuracy, one could include more terms from the Taylor series expansion. Another way is to decrease the value of h. However, decreasing h will increase the number of points to be solved, thereby increasing the length of computation and storage. Thus the step size has to be chosen by balancing accuracy requirements with computational loads. In addition, the step sizes hk do not need to be uniform at each step

Appendix G: Additional Details and Fortification for Chapter 7

G.5.1 Estimation of Local Truncation Error First, we need to estimate the truncation error at the kth step. Consider two integration methods: one that obtains an nth order approximation, and another that obtains an (n + 1)th order approximation. Starting from the same value of yk , let wk+1 and zk+1 be the update value for yk using the (n + 1)th and the nth order approximation methods, respectively, that is, for one-step methods wk+1 = yk + hk (tk , yk )|(n+1)th order

and

zk+1 = yk + hk (tk , yk )|(n)th order

where (tk , yk ) is a transition formula based on the particular method chosen. Subtracting zk+1 from wk+1 , we obtain an estimate of the truncation error of f (t, y), that is, τk+1 (hk ) =

|wk+1 − zk+1 | hk

(G.23)

In addition, we expect that the truncation error, τk+1 (hk ), is of the order of hnk , that is, for some constant C, τk+1 (hk ) = Chnk

→

Chnk =

|wk+1 − zk+1 | hk

(G.24)

= θhk ( θ > 0 ), such that the We want to find a different step size, hrevised k truncation error using the revised step size will be less than a prescribed tolerance , that is, τk+1 (θhk ) ≤ . Using (G.24), τk+1 (θhk ) = (C) (θhk )n = θn Chnk = θn

|wk+1 − zk+1 | ≤ hk

Rearranging,  θ≤

hk |wk+1 − zk+1 |

1/n (G.25)

To incorporate (G.25), we can set θ to be equal to the right hand side of (G.25) with  divided by 2, that is, 1/n  hk θˆ = (G.26) 2|wk+1 − zk+1 | This would guarantee a strict inequality in (G.25). The implementation of (G.26) is shown in the flowchart given in Figure G.1. In the flowchart, we see that if the truncation error, τk+1 , is less than the tolerance , we can set yk+1 to be wk+1 . Otherwise, we choose θ to be     ˆ (G.27) θ = min θmax , max θmin , θ If τk+1 happens to be much less than , the scaling factor θ will be greater than unity, which means the previous step size was unnecessarily small. Thus the step size could be increased. However, if τk+1 is greater than , θ will be less than unity, which means the step size has to be reduced to satisfy the accuracy requirements. As shown in the flowchart, we also need to constrain the step size hk to be within a preset maximum bound, hmax , and minimum bound, hmin .

725

726

Appendix G: Additional Details and Fortification for Chapter 7

k

k

hk

nth Order Method

(n+1)th Order Method

zk+1

wk+1

k+1

yk+1 tk+1 k

wk+1 (or zk+1) tk+hk

|wk+1 -zk+1| hk

=

yes k+1

k+1

no ^= min

hk

yes

1 2

1/n

max

1/n

k+1

max

^ min

min(hmax, hk)

hk

hmin ?

no

Figure G.1. Flowchart for error control.

G.5.2 Embedded Runge-Kutta Formulas The error control procedure shown in the flowsheet given in Figure G.1 requires two Runge-Kutta computations for the update of yk . One is an nth order method, whereas the other is an (n + 1)th order method. Normally, this would mean using two Runge-Kutta tableaus, one for each method. However, to improve efficiency, one could find a different tableau such that both updates can share some of the intermediate calculations of δij given in (7.13). This is done usually at a cost of increasing more terms in (7.13). Conversely, because both tableaus are merged into one tableau, the net change would usually mean fewer function evaluations. When two or more tableaus are merged to share the same function evaluations, we refer to these as embedded Runge-Kutta formulas, and the corresponding tableaus are called embedded Runge-Kutta tableaus. Two of the more popular embedded Runge-Kutta methods are the Fehlberg4-5 method and the Dormand-Prince-5-4 method. The Fehlberg tableau is given in (G.28). The row for zk+1 (second from the bottom) is used to determine the fourth-order update, whereas the row for wk+1 (last row) is used to determine the fifth-order update. However, the Fehlberg method uses zk+1 (the lower order result) as the update for yk+1 because the parameter values of the embedded

Appendix G: Additional Details and Fortification for Chapter 7

tableau were determined to minimize errors in the fourth-order estimates. The Dormand-Prince tableau is given in (G.29). The Dormand-Prince has a few more additional terms than the Fehlberg tableau. It was optimized for the fifth-order estimate instead. This means that the last row is the fourth-order estimate, whereas the second to the last row is the fifth-order estimate. So the Dormand-Prince tableau shown in (G.29) will use wk+1 , a fifth-order result, as the update for yk+1 . A MATLAB code for the Fehlberg 4/5 embedded Runge-Kutta method together with the error control algorithm shown in Figure G.1 is available on the book’s webpage as fehlberg45.m. ⎡

⎤

0

⎢ ⎢ ⎢ 1 ⎢ 4 ⎢ ⎢ ⎢ 3 ⎢ 8 ⎢ ⎢ ⎢ 12 ⎢ 13 ⎢ ⎢ ⎢ F45 : ⎢ ⎢ 1 ⎢ ⎢ ⎢ 1 ⎢ 2 ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ zk+1 ⎢ ⎢ ⎢ ⎣ wk+1 ⎡

1 4 3 32

9 32

1932 2197

− 7200 2197

7296 2197

439 216

−8

3680 513

845 − 4104

8 − 27

2

3544 − 2565

1859 4104

11 − 40

25 216

0

1408 2565

2197 4104

− 51

16 135

0

6656 12825

28561 56430

9 − 50

⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ 0 ⎥ ⎥ ⎥ ⎥ ⎦ 2 55

⎤

0

⎢ ⎢ ⎢ 1 ⎢ 5 ⎢ ⎢ ⎢ 3 ⎢ 10 ⎢ ⎢ ⎢ 4 ⎢ 5 ⎢ ⎢ ⎢ 8 ⎢ ⎢ 9 DP54 : ⎢ ⎢ ⎢ 1 ⎢ ⎢ ⎢ ⎢ 1 ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ zk+1 ⎢ ⎢ ⎢ ⎣ wk+1

(G.28)

1 5 3 40

9 40

44 45

56 − 15

32 9

19372 6561

− 25360 2187

64448 6561

− 212 729

9017 3168

− 355 33

46732 5247

49 176

5103 − 18656

35 384

0

500 1113

125 192

− 2187 6784

11 84

35 384

0

500 1113

125 192

− 2187 6784

11 84

5179 57600

0

7571 16695

393 640

92097 − 339200

187 2100

⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥(G.29) ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ 0 ⎥ ⎥ ⎥ ⎥ ⎦ 1 40

727

728

Appendix G: Additional Details and Fortification for Chapter 7

0.11

1

0.8

y

y1

0.6

2

0.4

0.2

0.1

0

0.2

0.4

0.6

0.8

0 0

1

0.2

t

0.4

0.6

0.8

1

t

Figure G.2. Numerical solution for Example G.1 showing varying step sizes based on errorcontrol strategy for tolerance  = 10−8 .

Consider the following set of differential equations to model the production of enzyme

EXAMPLE G.1.

dy1 dt dy2 dt μ

=

(μ − D) y1

=

D (y2f − y2 ) −

=

μmax y2 km + y2

μy1 Y

where Y = 0.4, D = 0.3, y2f = 4.0, μmax = 0.53, and km = 0.12 are the yield, dilution rate, feed composition, maximum rate, and Michaelis-Menten parameter, respectively. Assuming an initial condition of y(0) = (0.1, 0)T , we have the plots shown in Figure G.2 after applying the Fehlberg 4/5 embedded RungeKutta method using error control with tolerance  = 10−8 . We see that the step sizes are smaller near t = 0 but increased as necessary.

G.5.3 Step Doubling For implicit methods, such as the fourth-order Gauss-Legendre IRK method given in Section 7.2.2, there are no embedded methods. One approach is to use a higher order version and, together with the fourth-order result, obtain an estimate of the local error to be used for step-size control. Another method is the step-doubling approach. In this approach, one approximation, zk+2 ≈ yk+2 , is obtained by using the chosen implicit method twice with a step-size of hk . Another approximation, wk+2 ≈ yk+2 , is obtained by applying the chosen implicit method once, but with a step-size of 2hk . Let Err (hk ) be the local error using a step-size of hk , which will be proportional to hn+1 k , where n is the order of accuracy of the solver, then Err (hk ) = Chn+1 k

→

Err (2hk ) = 2n+1 Chn+1 k

Appendix G: Additional Details and Fortification for Chapter 7

and     wk+2 − zk+2 

=

2n+1 Chn+1 − 2Chn+1 k k  n+1  2 − 2 Err (hk )

= or Err (hk ) =

    w − z  k+2 k+2  2n+1 − 2

To control the error within a tolerance, , we need to change the step-size by a factor θ, that is,

θn+1

Err (θhk )

≤

C (θhk )n+1

≤

θn+1 Err (hk )     wk+2 − zk+2 

≤

2n+1 − 2

=



γ

where γ < 1, for example, γ = 0.9. This yields the formula for θ based on the stepdoubling approach: ⎛

  ⎞1/(n+1) γ 2n+1 − 2 ⎠ θ = ⎝   wk+2 − zk+2 

(G.30)

The MATLAB code for the Gauss-Legendre IRK is available on the book’s webpage as glirk.m, and it incorporates the error control based on the stepdoubling method.

EXAMPLE G.2.

Consider the van der Pol oscillator described by the following

equation: dy1 dt dy2 dt

=

y2

=

  μ 1 − y21 y2 − y1

subject to the initial condition y = (1, 1)T . When μ = 500, the system becomes practically stiff. Specifically, for the range t = 0 to t = 800, the Fehlberg 4/5 Runge Kutta will appear to “hang.” Instead, we could apply the Gauss-Legendre Implicit Runge Kutta, together with error-control based on the step-doubling approach using tolerance  = 10−6 . This results in the plot shown in Figure G.3, which shows that small step sizes are needed where the slopes are nearly vertical.

729

730

Appendix G: Additional Details and Fortification for Chapter 7

3

2

1

y1

Figure G.3. The response for a van der Pol oscillator when μ = 500 using Gauss-Legendre IRK method with error control based on step-doubling procedure.

0

−1

−2

−3 0

200

400

600

800

t

G.6 Proof of Solution of Difference Equation, Theorem 7.1 First, we can rewrite the (7.46) in terms of constants β j, instead of c j, as follows: ⎞ ⎛ ⎛ ⎞ k j −1 k j −1   n! ⎠ (σ j )n c j, n  ⎠ (σ j )n = ⎝ β j, S (j, n) = ⎝ (n − )! =0

k j −1



=  where Dσ j =

d



=0

  β j, (σ j ) Dσ j (σ j )n

=0

. Next, apply the difference operators of (7.43) on S (j, n) in d (σ j ) p place of y, with χ(σ j ) = i=0 αi (σ j )i , 8 p 9 k j −1 p        n+i i = αi Q S (j, n) β j, (σ j ) Dσ j αi (σ j ) =0

i=0

k j −1



=

i=0

β j, (σ j ) Dσ j



χ(σ j ) (σ j )n



=0 k j −1



=

β j, (σ j )

=0

  m=0

Because σ j is a k j -fold root of χ(σ) = 0,   ; Dσ j χ(σ j ) = 0 →

p 

    ! −m Dm (σ j )n σ j χ(σ j ) Dσ j m!( − m)!

 = 0, 1, . . . , k j − 1

αi Qi (S (j, n)) = 0

i=0

Combining all the results, p  i=0

αi Q (yn ) = i

 p M   j =1

i=0

   =0 αi Q S (j, n) i

Appendix G: Additional Details and Fortification for Chapter 7

G.7 Nonlinear Boundary Value Problems Consider the nonlinear boundary value problems given by d x = F (t, x) dt subject to the nonlinear boundary conditions,

(G.31)

q (x(0), x(T )) = 0

(G.32)

First, let us define the following vectors: 1. Let x0 be any initial value of x for the system given in (G.31). 2. Let xT be the value of x at t = T corresponding to x0 . Thus xT = xT (x0 )

(G.33)

and these vectors could be evaluated by using any initial value solvers such as Runge-Kutta method after setting x0 as the initial condition. The main idea of the shooting method is to find the appropriate value for x0 such that the boundary conditions given in (G.32) are satisfied, that is, Find x0

q (x0 , xT (x0 )) = 0

such that

For some small problems, a trial-and-error approach may be sufficient. However, as the number of variables and the level of complexity increase, a systematic method such as Newton’s method is preferable.4 (0) Newton’s method uses an initial guess, x0 , and improves the value of x0 iteratively using the following update equation: (k+1)

x0 where, (k)

=

J

=

x0

(k)

(k)

= x0 + x0

   (k) (k) −J −1 q x0 , xT x0  dq  dx0 x0 =x(k) 0

(G.34)

(G.35) (G.36)

   (k) (k) (k) is close to zero, we can set x0 = x0 and solve for x(t) from Once q x0 , xT x0 t = 0 to t = T . If the number of iterations exceeds a maximum, then either a better initial guess is required or a different method needs to be explored. The terms in (G.36) generate a companion set of initial value problem. Specifically, J is the square Jacobian matrix of q. The added complexity stems from the dependence of q on xT , which in turn depends on x0 through the integration process of (G.31). Let the boundary conditions be given as ⎞ ⎛ q1 (x01 , . . . , x0n , xT 1 , . . . , xTn ) ⎟ ⎜ .. (G.37) q (x0 , xT ) = ⎝ ⎠=0 . qn (x01 , . . . , x0n , xT 1 , . . . , xTn ) 4

The Newton-search approach is not guaranteed to converge for all systems. It is a local scheme and thus requires a good initial guess.

731

732

Appendix G: Additional Details and Fortification for Chapter 7

then dq dx0

 = =

∂q (a, b) ∂a ∂a ∂x0





a=x0 ,b=xT

+

∂q (a, b) db ∂b dx0

 a=x0 ,b=xT

Qa (x0 , xT ) + Qb (x0 , xT ) M (T )

(G.38)

where, ⎛ =

Qa

ηij

=

Qb

=

⎞ η1n .. ⎟ . ⎠ ηnn

··· .. . ···

η11 ⎜ .. ⎝ . ηn1

(G.39)

 ∂qi (a1 , . . . , an , b1 , . . . , bn )   ∂a j ak =x0k ,b =xT ⎞ ⎛ ω11 · · · ω1n ⎜ .. .. ⎟ .. ⎝ . . . ⎠ ···

ωn1

(G.40)

(G.41)

ωnn

ωij

=

 ∂qi (a1 , . . . , an , b1 , . . . , bn )   ∂bj ak =x0k ,b =xT

M(T )

=

dxT dx0

(G.42) (G.43)

To determine M(T ), take the derivative of the original differential equation (G.31) with respect to x0 ,   d d d x = F (t, x) dx0 dt dx0   d dx ∂F dx = dt dx0 ∂x dx0 d M(t) dt

=

A (t, x) M(t)

(G.44)

where M(t)

=

dx dx0

(G.45)

A(t, x)

=

∂F ∂x

(G.46)

and M(0) = I

(G.47)

Note that A(t, x) depends on the x consistent with the x0 used. Thus the following integration needs to be performed simultaneously: d x dt d M dt

=

F(t, x)

=

A (t, x) M

x(0) = x0 M(0) = I

(G.48) (G.49)

Appendix G: Additional Details and Fortification for Chapter 7

x0

dx dt dM dt

(t, )

x(0)=x0

t

M(0)=I

MT=M(T)

q = q(x0,xT)

xT=x(T)

Qa (x0,xT) Qb (x0,xT)

no

q

yes

J=Qa+QbMT x0

x0

-1

J q

dx dt

(t, )

x(0)=x0

x0 + x 0

Solution: x(0),x(h),x(2h),...,x(T) Figure G.4. A flowchart for nonlinear shooting implemented with Newton’s method.

Having calculated xT = x(T ) and M(T ), we can then substitute these values together with x0 to determine Qa , Qb and dq/dx0 . Thereafter, the update to x0 can be determined, and the iteration continues until the desired tolerance on q is obtained. A flowchart showing the calculation sequences is given in Figure G.4.

EXAMPLE G.3.

Consider the following set of differential equations: ⎞ ⎛ ⎞ ⎛ −k1 e−2t x1 x2 + k3 x d ⎝ 1 ⎠ ⎝ = x2 −k1 e−2t x1 x2 + k2 x3 ⎠ dt x3 k1 e−2t x1 x2 − k2 x3

subject to the following boundary conditions: ⎛ ⎞ x1 (0) − x2 (T ) − 0.164 q (x(0), x(T )) = ⎝ x2 (0)x2 (T ) − 0.682 ⎠ = 0 x3 (0) + x3 (T ) − 1.136 with T = 2, k1 = 10, k2 = 3 and k3 = 1.

733

734

Appendix G: Additional Details and Fortification for Chapter 7

1.5

x2 1.0

x1

x 0.5

0

0

Figure G.5. Solution for boundary value problem given in Example G.3 .

x3

0.5

1

1.5

2

t

We can calculate Qa and Qb to be in a form that can be evaluated readily based on values of x0 and xT , ⎛ ⎞ 1 0 0 Qa = ⎝ 0 x2 (T ) 0 ⎠ 0 0 1 ⎛ ⎞ 0 −1 0 Qb = ⎝ 0 x2 (0) 0 ⎠ 0 0 1 Similarly, we can calculate A = ∂F/∂x,  ⎛  −2t −k1 e−2t x2  A(t, x) = ⎝ −k1 e x2 k1 e−2t x2

  −2t −k1 e−2t x1  −k1 e−2t x1 k1 e x1

⎞ 0 k2 ⎠ −k2

(0)

Using an initial guess of x0 = (1, 1, 1)T and a tolerance of  = 1 × 10−10 , it took five iterations to converge to the following initial and final conditions: ⎛ ⎛ ⎞ ⎞ 1.516 1.083 xT = ⎝ 1.352 ⎠ x0 = ⎝ 0.504 ⎠ 0.992 0.144 Plots of the solutions are shown in Figure G.5. (A MATLAB file nbvp.m is available on the book’s webpage and solves this specific example. The code contains sections that are customizable to apply to different nonlinear boundary value problems.)

G.8 Ricatti Equation Method Consider the linear differential equation, d x = A(t)x + b(t) dt

(G.50)

Appendix G: Additional Details and Fortification for Chapter 7

with separated boundary conditions such that k conditions are specified at t = 0 and (n − k) conditions are specified at t = T , Q0 x(0)

=

β0

(G.51)

QT x(T )

=

βT

(G.52)

where Q0 is a k × n matrix of constants and QT is an (n − k) × n matrix of constants. As an alternative to the shooting method, we look for a transformation of the original state variable given by x(t) = S(t)z(t)

(G.53)

where S(t) is an n × n transformation matrix and z(t) is the new state vector. The aim of the transformation is to recast the original problem into a partially decoupled problem such that the solution of first k values of z can be solved independently of the last (n − k) values of z, that is,        d z1 H11 (t) q1 (t) 0 z1 (G.54) = + z2 z2 H21 (t) H22 (t) q2 (t) dt where z1 (t)[=]k × 1 and z2 (t)[=](n − k) × 1. In addition, the transformation will be done such that the z1 (0) can be specified from (G.51), whereas z2 (T ) can be specified from (G.52). Thus z1 is first solved using initial value solvers to determine z1 (t = T ). Afterward, z1 (T ) is combined with z2 (T ), after using (G.52), to form z(t = T ). The terminal condition for x(t) at t = T can then be found from (G.53). Having x(T ), the trajectory of x(t) can be evaluated by integrating backward from t = T to t = 0. To obtain the form in (G.54), we first apply (G.53) to the original equation, (G.50),   d d S z + S z = ASz + b dt dt   d d −1 z = S (t) A(t)S(t) − S z + S−1 (t)b(t) dt dt = where

H(t)z + q(t)

  d S (t) A(t)S(t) − S dt

=

H(t)

d S dt

=

A(t)S(t) − S(t)H(t)

−1

(G.55)

and q(t) = S−1 b(t)

(G.56)

We can choose S to be an upper triangular matrix given by   Ik R(t) S= 0 In−k whose inverse is given by −1

S

 =

Ik 0

−R(t) In−k

(G.57)

 (G.58)

735

736

Appendix G: Additional Details and Fortification for Chapter 7

After substitution of (G.57) into (G.55),        H11 A11 A12 I R(t) I R(t) 0 dtd R = − A21 A22 0 I 0 I H21 0 0   A11 − (H11 + RH21 ) A11 R + A12 − RH22 = A21 − H21 A21 R + A22 − H22

0 H22



By comparing elements on both sides, we have the following equations H21

=

A21

H22

=

A21 R + A22

H11

=

A11 − RH21 = A11 − RA21

d R dt

=

A11 R + A12 − RA21 R − RA22

(G.59)

where the last equation is a matrix Ricatti equation. Because H11 depends on R(t), we need to solve for z1 and R using the first k equations of (G.54) and (G.59) as initial value problems, that is, d z1 dt d R dt

=

(A11 − RA21 ) z1 + q1

=

A11 R + A12 − RA21 R − RA22

(G.60)

Note that z1 is a vector, whereas R is a matrix. To determine the required initial conditions, we can find z1 (0) in terms of R(0) using (G.53) and (G.58),   z1 (0) = Ik −R x(0) (G.61) Assume that the first k columns of Q0 in (G.51) are linearly independent5 ; that is, let C be the nonsingular matrix consisting of the first k columns of Q0 , then   Q0 x(0) = C I C−1 D x(0) = β0 Next, choose R(0) = −C−1 D and premultiply z1 (0) (in (G.61)) by C,   Cz1 (0) = C I C−1 D x(0) = β0 → z1 (0) = C−1 β0

(G.62)

In summary, the first phase, known as the forward-sweep phase of the Ricatti equation method, is to solve for R(t) and z1 (t), that is, d R = A11 R + A12 − RA21 R − RA22 dt   C D , followed by where Q0 = d z1 = (A11 − RA21 ) z1 + dt

R(0) = −C−1 D

;



 I

−R

b

;

z1 (0) = C−1 β0

and integrate until t = T to obtain the values of z1 (T ) and R(T ). 5

If the first k columns of Q0 are not invertible, a reordering of x may be required.

(G.63)

(G.64)

Appendix G: Additional Details and Fortification for Chapter 7

The second phase of the method is to find the conditions for x(T ) by combining the results from the first phase with the other set of boundary conditions given by (G.52). By partitioning QT as   F G QT = where F is (n − k) × n and G is (n − k) × (n − k), we get     I R(T ) z1 (T ) F G QT x(T ) = z2 (T ) 0 I F z1 + FR(T )z2 (T ) + Gz2 (T ) z2 (T ) = (FR(T ) + G)

−1

(βT − F z1 (T ))

which can be used to form x(T ), that is,    I R(T )  x(T ) = z(T ) 0 I

=

βT

=

βT (G.65)

(G.66)

Having evaluated x(T ) means we now have all the information at one boundary. We could then use the original differential equations given in (G.50) and integrate backward starting from t = T until t = 0. This second phase is also known as the backward-sweep phase of the Ricatti equation method. The Ricatti equation method (which is also sometimes called the invariant embedding method) is sometimes more stable than the shooting method, especially when the process (7.55) is unstable. However, there are also situations when the shooting methods turn out to be more stable. Thus both methods may need to be explored in case one or the other does not yield good results. Note also that our development of the Ricatti equation method is limited to cases with separated boundary conditions.

737

APPENDIX H

Additional Details and Fortification for Chapter 8

H.1 Bifurcation Analysis The behavior around a non-hyperbolic equilibrium point can change under slight modifications of the process parameters. Under these conditions, the system is classified as structurally unstable. By perturbing the parameter slightly, the characteristics can sometimes yield additional equilibrium points and can change the stability of equilibrium points. Bifuraction analysis is the study of how the structural behaviors of the system are affected by variations in the key parameters. For the one-dimensional case, there are three main types of bifurcations. A summary of the different types of bifurcations for one-dimensional systems is given in Table H.1. Included in the table are the normal forms and the corresponding bifurcation diagram. The bifurcation diagrams show the locus of equilibrium points, if they exist, at different values of parameter r. We use the convention that represents the locus of stable equilibrium points by solid curves and the locus of unstable equilibrium points by dashed curves. The first type of bifurcation is the saddle-node. Saddle-node bifurcations are characterized by the absence of equilibrium points to one side of the non-hyperbolic equilibrium point, and saddle-node bifurcations are also known as blue-sky bifurcations to highlight the sudden appearance of equilibrium points as if they appeared “out of the sky.” The term “saddle-node” is more appropriate for the 2D case. The second type of bifurcation is the transcritical bifurctation. Transcritical bifurcations are characterized by the intersection of two locus of equilibrium points at a non-hyperbolic point. After both curves cross each other, their stability switch from stable to unstable and vice versa. The third type of bifurcation is the pitchfork bifurcation. Pitchfork bifurcations are characterized by additional equilibrium points as they cross the non-hyperbolic equilibrium point from a single locus curve of stable (supercritical) or unstable (subcritical) equilibrium points. The name of this bifurcation comes from the bifurcation diagram (as shown in Table H.1) resembling a pitchfork. For cases that are more general than the given normal forms, let x˙ = f (x, r) where x = 0 is a non-hyperbolic equilibrium point at r = 0. A Taylor series expansion around (x, r) = (0, 0) is given by f (x, r) = f (0, 0) + x 738

∂f x2 ∂ 2 f r2 ∂ 2 f ∂2 f ∂f +r + + + rx + ··· ∂x ∂r 2 ∂x2 2 ∂r2 ∂r∂x

Appendix H: Additional Details and Fortification for Chapter 8

739

Table H.1. Types of bifurcations for one-dimensional systems

1

Type

Normal form

Saddle-Node

x˙ = r + x2

Bifurcation diagram 1

unstable

x

0

stable −1 −1

2

Transcritical

x˙ = rx − x2

0

1

r

1

stable

x

stable 0

unstable

unstable −1 −1

0

1

r 3

Pitchfork

x˙ = rx − x3

(Supercritical) 1

x0

stable

unstable

stable

stable −1 −1

0

1

r x˙ = rx + x3

(Subcritical) 1

unstable

x0

stable

unstable

unstable −1 −1

0

r

1

where all the various partial derivatives are evaluated at (x, r) = (0, 0). Because (x, r) = (0, 0) is a non-hyperbolic equilibrium point, the first two terms are zero, that is, f (0, 0) = 0 and ∂f/∂x(0, 0) = 0. We will truncate the series after the second-order derivatives to yield bifurcation analysis of saddle-node bifurcations and transcritical bifurcations. This means that equilibrium points near (x, r) = (0, 0) will be given by the roots of the second-order polynomial in x, α2 (r) x2 + α1 (r) x + α0 (r) = 0

(H.1)

740

Appendix H: Additional Details and Fortification for Chapter 8

where α2 (r)

=

1 ∂2 f 2 ∂x2

α1 (r)

=

r

∂2 f ∂r∂x

α0 (r)

=

r

∂f r2 ∂ 2 f + ∂r 2 ∂r2

which was obtained by setting the right-hand side of the Taylor series expansion to zero. Solving for the roots of (H.1), we find the neighboring equilibrium points around (x, r) = (0, 0), 6  2 2  2   ∂2 f ∂ f 1∂ f ∂f r2 ∂ 2 f 2 −r ± r −4 r + ∂r∂x ∂r∂x 2 ∂x2 ∂r 2 ∂r2 xeq = (H.2) 2 ∂ f ∂x2 For saddle-node bifurcations, consider |r|  1. Then (H.2) will reduce to 6    2 −1  ∂f ∂ f  xeq saddle−node = ± −2r (H.3) ∂r ∂x2 which then requires 

∂f r ∂r



∂2 f ∂x2

−1

0 and  0 = (H.6) ∂r∂x ∂x2 ∂r2 ∂x2 As r changes sign, the stability of the equilibrium points will switch, thereby giving the character of transcritical bifurcations. For both saddle-node and transcritical bifurcations, the stability can be assessed by regrouping the Taylor series approximation as   x˙ ≈ α0 (r) + α1 (r) + α2 (r)x x = α0 (r) + β(x,r) x where β(x,r) = α1 (r) + α2 (r)x

Appendix H: Additional Details and Fortification for Chapter 8

741

2 xeq

1 0

Figure H.1. Two-parameter bifurcation diagram.

-1 -2 -2 r

0

1 0 2

-1

Then applying the formulas for xeq (Equation (H.3) for saddle-node bifurcations and Equation (H.5) for transcritical bifurcations), we find that xeq,i

is stable if

β(xeq,i ,r) < 0

for i = 1, 2

For pitchfork bifurcations, the Taylor series will need to include third-order derivatives such that a third-order polynomial can be obtained for the equilibrium points. The computations are lengthier, but with the additional condition that ∂f/∂r(0, 0) = 0 and ∂ 2 f/∂x2 (0, 0) = 0, the conditions simplify to the following conditions ⎧ ⎨ > 0 for single equilibrium points 2 3 ∂ f ∂ f r (H.7) ⎩ < 0 for three equilibrium points ∂x∂r ∂x3 It is important to remember that all the partial derivatives given in the conditions (H.4), (H.6), and (H.7) are evaluated at (x, r) = (0, 0). Aside from the three type of bifurcations discussed thus far, the introduction of one more parameter can also make the bifurcations change, including the addition or removal of non-hyperbolic equilibrium points. This situation is known as codimension two bifurcations. An example of these types of bifurcation is the catastrophe model given by x˙ = f (x, r, h) = x3 − rx − h

(H.8)

where r and h are parameters. A surface locus of equilibrium points is shown in Figure H.1. In the figure, we see that the surface has a continuous fold, and thus, dependent on the values of r and h, there can be either one, two, or three equilibrium points. These regions can be separated by two intersecting curves as shown in the (r, h) plane as shown in Figure H.2. The point where the two separating curves intersect is known as the cusp point. Many physical phenomenon, such as phase changes of material, that is, vapor liquid equilibria, are described by these types of bifurcations or catastrophe models. Next, consider the bifurcation diagram for xeq at r = 2 as shown in Figure H.3. When r = 2, there are two non-hyperbolic equilibrium points: one at (x, h) = (0.816, −1.089) and another at (x, h) = (−0.816, 1.089), both of which yield

h

742

Appendix H: Additional Details and Fortification for Chapter 8 2

1.5

one equilibrium point

1

0.5

h

three equilibrium points

0

−0.5

Figure H.2. Phase diagram in the (r, h)-plane.

cusp point

−1

−1.5

−2 −2

−1

0

1

2

r saddle-node bifurcations. When h > −1.089 and gradually decreased, the equilibrium points following the top curve in Figure H.3 will also decrease continuously. However, as h moves past the critical value h = −1.089, the equilibrium point will jump to follow the values of the lower curve. The opposite thing happens for the lower curve; that is, as h gradually increases until it passes the value of 1.089, the equilibrium point jumps to follow the upper locus of equilibrium points. This characteristic of having the behavior depend on the direction of parameter change is known as hysteresis. The bifurcations of second-order systems include all three types of the first-order cases, namely saddle-node, transcritical, and pitchfork bifurcations. These three types of bifurcations are extended by means of simply adding one more differential equation. The canonical forms are given in Table H.2. These types of bifurcations are centered at non-hyperbolic equilibrium points that have zero eigenvalues. The Hopf bifurcation is a type of bifurcation that is not available to onedimensional systems because it involves pure imaginary eigenvalues. These bifurcations yield the appearance or disappearance of limit cycles. A supercritical Hopf bifurcation occurs when a stable focus can shift to a stable limit cycle. Conversely, a subcritical Hopf bifurcation occurs when an unstable limit cycle changes to an 2 stable

1.5

1

0.5

x

unstable

0

Figure H.3. Bifurcation diagram for (x, h) when r = 2.

−0.5

−1

−1.5 stable

−2 −2

−1

0

h

1

2

Appendix H: Additional Details and Fortification for Chapter 8 Table H.2. Normal forms for Bifurcations of 2D Systems Type

Normal form

1

Saddle-Node

y˙ = −y x˙ = r + x2

2

Transcritical

y˙ = −y x˙ = rx − x2

3

Pitchfork

Supercritical: y˙ = −y x˙ = rx − x3 Subcritical: y˙ = −y x˙ = rx + x3

4

Hopf

θ˙ = ω ρ˙ = μρ + aρ3 Supercritical: a < 0 Subcritical: a > 0

unstable focus. The canonical form of a Hopf bifucation given in terms of polar coordinates (ρ, θ), dθ =ω dt

and

dρ = μρ + aρ3 dt

! where ρ = x2 + y2 and θ = tan−1 (y/x). It can be shown that when a < 0, the system exhibits a supercritical Hopf bifurcation. However, when a > 0, the system exhibits a subcritical Hopf bifurcation. These are shown in Figures H.4. It turns out that Hopf bifurcations can occur for orders ≥ 2. A general theorem is available that prescribes a set of sufficient conditions for the existence of a Hopf bifurcation. Let λh be a value of parameter λ such that the system dx/dt = f (x; λ) has an equilibrium point xeq (λh ) with the Jacobian matrix √ J = df/dx at x = xeq (λh ) having a pair of pure imaginary eigenvalues, ±i μ(λh ) (i = −1), whereas the rest of the eigenvalues have nonzero real parts. In addition, let the real and imaginary parts of the eigenvalues μ(λ) be smooth functions of parameter λ in which  d  Re(μ(λ)) = 0 dλ

THEOREM H.1.

in a neighborhood around λh . Under these conditions, the system will have a Hopf bifurcation at λ = λh . There are several physical systems that exhibit Hopf bifurcations, such as in the fields of biomedical science, aeronautics, fluid mechanics, and chemistry.1 In 1

A good elementary treatment of Hopf bifurcations, including several examples and exercises, can be found in S. Strogatz, Nonlinear Dynamics and Chaos, Perseus Book Publishing, Massachusetts, 1994.

743

744

Appendix H: Additional Details and Fortification for Chapter 8

(a) Supercritical Hopf bifurcations.

(b) Subcritical Hopf bifurcations. Figure H.4. Phase plane plots showing supercritical and subcritical Hopf bifurcations. (ω = 0.5).

chemistry, there are several well-known reaction systems, such as the BelousovZhabotinsky (BZ) system, known collectively as oscillating chemical reactions. Depending on the critical conditions, the systems can oscillate spontaneously. One of the well-known examples of a Hopf bifurcation is the Brusselator reaction, which is given in Exercise E8.19. Although it is strictly fictitious, its simplification still allows one to understand the onset of Hopf bifurcations in real systems.

APPENDIX I

Additional Details and Fortification for Chapter 9

I.1 Details on Series Solution of Second-Order Systems For N = 2, the differential equation for which x = 0 is a regular singular point is given by x2 P˜ 2 (x)

d2 y dy + xP˜ 1 (x) + P˜ 0 (x)y = 0 dx2 dx

& P2 (x)

=

& ρ2,0 + & ρ2,1 x + & ρ2,2 x2 + · · ·

& P1 (x)

=

& ρ1,0 + & ρ1,1 x + & ρ1,2 x2 + · · ·

& P0 (x)

=

& ρ0,0 + & ρ0,1 x + & ρ0,2 x2 + · · ·

(I.1)

where

(I.2)

and & ρ2,0 = 0. The indicial equation (9.28) becomes & ρ0,0 + & ρ1,0 r + & ρ2,0 (r)(r − 1)

=

0

& ρ2,0 r2 + (& ρ1,0 − & ρ2,0 ) r + & ρ0,0

=

0

and the indicial roots are r=

ρ1,0 ) ± ρ2,0 − & (&

"

ρ1,0 )2 − 4& ρ0,0& ρ2,0 ρ2,0 − & (& 2& ρ2,0

(I.3)

(I.4)

We denote the larger root (if real) by ra and the other root by rb. When the roots differ by an integer, say ra − rb = m ≥ 0, ra + r b

=

2ra − m

=

ra

=

1−

& ρ1,0 & ρ2,0

  & ρ1,0 1 m+1− 2 & ρ2,0

(I.5)

When the roots are equal, m = 0,

  & ρ1,0 1 ra = 1− 2 & ρ2,0

(I.6) 745

746

Appendix I: Additional Details and Fortification for Chapter 9

Using ra , we are guaranteed one solution, which we will denote by u(x), u(x) =

∞ 

& φn (ra )xra +n

(I.7)

n=0

where

& φn (ra )

Qn,k (ra )

=

=

⎧ ⎪ ⎪ ⎪ ⎨ 1 ⎪ ⎪ ⎪ ⎩ −

if n = 0 n−1 k=0

Qn,k (ra )& φk (ra ) Qn,n (ra )

if n > 0

& ρ0,n−k + & ρ1,n−k (k + ra ) + & ρ2,n−k (k + ra )(k + ra − 1)

If (ra − rb) is not an integer, the second solution, v(x), is immediately given by v(x) =

∞ 

& φn (rb)xrb+n

(I.8)

n=0

where

& φn (rb)

Qn,k (rb)

=

=

⎧ ⎪ ⎪ ⎪ ⎨ 1 ⎪ ⎪ ⎪ ⎩ −

if n = 0 n−1 k=0

Qn,k (rb)& φk (rb) Qn,n (rb)

if n > 0

& ρ0,n−k + & ρ1,n−k (k + rb) + & ρ2,n−k (k + rb)(k + rb − 1)

If the indicial roots differ by an integer, that is, m ≥ 0, we can use the d’Alembert method of order reduction (cf. Lemma I.1 in Section I.2) to find the other solution. For N = 2, this means the second solution is given by  v(x) = u(x) z(x)dx (I.9) where z(x) is an intermediate function that solves a first-order differential equation resulting from the d’Alembert order reduction method. Using u(x) as obtained in (I.7), z(x) can be obtained by solving   dz du 2& 2& & + 2x P2 (x) + xP1 (x)u z = 0 x P2 (x)u dx dx   & P1 (x) 1 dz 1 du = − +2 (I.10) z dx u dx x& P2 (x) P1 (x) defined by equations (I.7) and (I.2), respectively, the leftWith u, & P2 (x) and & hand side of (I.10) can be replaced by an infinite series, ∞  1 dz =− (αn + βn ) xn z dx n=−1

(I.11)

Appendix I: Additional Details and Fortification for Chapter 9

where the terms αn and βn are defined as ⎧ ⎪ ⎨ 2ra   αn =  ⎪ & φ φn+1 (ra ) − n−1 α (r ) ⎩ 2(ra + n + 1)& k n−k a k=−1

βn

=

⎧ ⎪ ρ1,0 /& ρ2,0 ⎨ &   n−1 ⎪ & ρ ρ2,0 − β & ρ ⎩ 1,n+1 k=−1 k 2,n−k /&

747

if n = −1 if n ≥ 0

(I.12)

if n = −1 if n ≥ 0

(I.13)

For (I.12), we used the fact that & φ0 (ra ) = 1. For indicial roots differing by an integer, we can use (I.5), and the coefficient for first term involving (1/x) in (I.11) becomes α−1 + β−1 =

& ρ1,0 + 2ra = m + 1 & ρ2,0

Then returning to (I.11), z can be evaluated as follows:   ∞ 1 dz m+1  n = − + (αn + βn ) x z dx x n=0   ∞  m+1   (αn + βn ) n+1 ln(z) = − ln x + x n+1 n=0 8 ∞ 9  (αn + βn ) −(m+1) n+1 z = x exp − x n+1 n=0

We can also expand the exponential function as a Taylor series, 8 ∞ 9  (αn − βn ) n+1 exp − x = γ0 + γ1 x + γ2 x2 + · · · n+1 n=0

Due to the complexity of the definitions of γi , i = 1, 2, . . ., we just treat the γi ’s as constants for now. The Taylor series expansion is being used at this point only to find the form needed for the second independent solution. Once the solution forms are set, a direct substitution is used later to find the unknown coefficients. Thus we can rewrite z as ⎧ m−1 k−m−1 ⎪ k=0 γk x ⎪  ⎨ −1 n−m−1 + γm x + ∞ if m > 0 n=m+1 γn x z= ⎪ ⎪  ⎩ n−1 γ0 x−1 + ∞ if m = 0 n=1 γn x and ⎧ m−1 k−m ⎪ k=0 (γk /k − m) ⎪  x∞ ⎨ + γm ln |x| + n=m+1 (γn /n − m) xn−m zdx = ⎪ ⎪  ⎩ n γ0 ln |x| + ∞ n=1 (γn /n) x

if m > 0 if m = 0

748

Appendix I: Additional Details and Fortification for Chapter 9

This integral can now be combined with u to yield the form for the second independent solution, that is,  v(x) = u(x) zdx =

∞ 

& φn x

ra +n

 zdx

n=0

v(x)

=

⎧ ∞ ⎨ ηu ln |x| + n=0 bn xrb+n ⎩

u ln |x| +

∞

rb +n n=1 bn x

if m > 0 (I.14) if m = 0

Note that for m = 0, the infinite series starts at n = 1 and the coefficient of (u ln |x|) is one. The parameter η is set equal to 1 when m = 0 because η will later be combined with a constant of integration. However, when m > 0, η should not be fixed to 1, because η = 0 in some cases. Instead, we will set b0 = 1 in anticipation of merging with the arbitrary constant of integration. Having found the necessary forms of the second solution, Theorem 9.2 summarizes the general solution of a second-order linear differential equation that includes the recurrence formulas needed for the coefficients of the power series based on the Frobenius method.

I.2 Method of Order Reduction For an Nth -order homogenous linear differential equation given by N 

i (x)

i=0

di y =0 dxi

(I.15)

Suppose we know one solution, say, u(x), that solves (I.15). By introducing another function, q(x), as a multiplier to u(x), we can obtain y = q(x)u(x)

(I.16)

as another solution to (I.15) that is linearly independent from u. To evaluate q(x), we will need to solve another linear differential equation of reduced order as given in the following lemma: d’Alembert’s Method of Order Reduction Let q(x) be given by  q(x) = z(x)dx

LEMMA I.1.

(I.17)

where z(x) is the solution of an (N − 1)th order differential equation given by N 

F i (x)

i=1

di−1 z =0 dxi−1

(I.18)

with F i (x) =

N  k=i

k! d(k−i) u k (x) (k−i) (k − i)!i! dx

(I.19)

Appendix I: Additional Details and Fortification for Chapter 9

and u(x) is a known solution of (I.15). Then y = q(x)u(x) is also a solution of (I.15). PROOF.

First, applying Leibnitz’s rule (9.6) to the n th derivative of the product y = qu,  j i  di y  d q d(i−j ) u i = j dxi dx j dx(i−j ) j =0

where



i j



i! j !(i − j )!

=

Substituting these derivatives into (I.15), N 

 q

N  i=0

di u i (x) i dx

i (x)

+

N 

=

0

 j i   d q d(i−j ) u i j dx j dx(i−j )

=

0

j =0

i=0



 j i   d q d(i−j ) u i j dx j dx(i−j )

i (x)

i=1

j =1

Because u satisfies (I.15), the first group of terms vanishes. The remaining terms can then be reindexed to yield  N   N   k  d(k−i) u di q k (x) (k−i) =0 i dxi dx i=1

k=i

Letting z = dq/dx, we end up with an (N − 1)th order linear differential equation in z.

This method can be used repeatedly for the reduced order differential equations. However, in doing so, we require that at least one solution is available at each stage of the order reductions. Fortunately, from the results of the previous section, it is always possible to find at least one solution for the differential equations using the Frobenius method. For instance, with N = 3, the Frobenius series method will generate one solution, say, u. Then via d’Alembert’s method, another solution given by y = qu produces a second-order differential equation for z = dq/dt. The Frobenius series method can generate one solution for this second-order equation, say, v. Applying the order reduction method one more time for z = wv, we end up with having to solve a first-order differential equation for w.1 Having solved for w, we can go backward:

1

z

=

q

=

α1 v + α2 wv   α1 vdx + α2 wv dx

y

=

β1 u + β2 qu

=

β1 u + β2 α1 u



 vdx + β2 α2 u

wv dx

The resulting first-order differential equation is always of the separable type.

749

750

Appendix I: Additional Details and Fortification for Chapter 9

with α1 , α2 , β1 , and β2 as arbitrary coefficients. Thus the approach of recursive order reduction can be used to generate the general solution for homogenous linear differential equation. One disclaimer to this solution approach is that, although the general solutions can be found in principle, the evaluation of the integrals via quadrature may still be difficult. This means that in case another simpler method is available, such as when all the indicial roots are distinct, those approaches should be attempted first.

I.3 Examples of Solution of Regular Singular Points In this section, we have three examples to show how Theorem 9.2, which is the Frobenius series solution to linear second-order equations, is applied to the cases where ra − rb is not an integer, ra − rb = 0, and ra − rb = m is a positive integer.

EXAMPLE I.1.

Given the equation

d2 y dy + x(1 − x) −y=0 dx2 dx The terms for & ρi,j are & ρ2,0 = 2, & ρ1,0 = 1, & ρ1,1 = −1, and & ρ0,0 = −1, whereas the rest are zero. The indicial roots become ra = 1 and rb = −0.5. Because the φn (rb). The only nonzero values of difference is not an integer, η = 0 and bn = & Qn,k are 2x2

Qn,n (r) = n (2n + 4r − 1)

Qn,n−1 (r) = − (n − 1 + r)

and

The recurrence formulas are then given by & φn (r) =

n−1+r & φn−1 (r) n (2n + 4r − 1)

Thus & φn (ra )

= =

& φn (rb)

=

1 & φn−1 (ra ) = 2n + 3



1 2n + 3



where

n>0

   1 1 & ··· φ0 2n + 1 5

(2n + 2)(2n) · · · 6 · 4! 2n+1 (n + 1)! =3 (2n + 3)(2n + 2)(2n + 1) · · · 5 · 4! (2n + 3)!      1 1 1 1 1 & φn−1 (rb) = ··· φ0 (rb) = n 2n 2n 2(n − 1) 2 2 n!

and the complete solution is given by y(x) = A

∞ ∞   2n+1 (n + 1)! n+1 1 n−(1/2) x 3 x +B (2n + 3)! 2n n! n=0

n=0

This can be put in closed form as follows:   7 x 3 2π x/2 ex/2 y(x) = A −3 + e erf + B√ 2 x 2 x

Appendix I: Additional Details and Fortification for Chapter 9 EXAMPLE I.2.

Given the equation

d2 y dy +x + xy = 0 dx2 dx The terms for & ρi,j are & ρ2,0 = 1, & ρ1,0 = 1, and & ρ0,1 = 1, whereas the rest are zero. The indicial roots are ra = rb = 0. Because the difference is an integer with m = 0, we have η = 1. The only nonzero values of Qn,k are x2

Qn,n (0) = n 2 Thus & φ0 (0) = 1 and for n > 0, & φn (0) = −

1 & φn−1 (0) = n2

Qn,n−1 (0) = 1

and 

−1 n2



−1 (n − 1)2

 · · · (−1) =

(−1)n (n!)2

which yields the first solution u(x) u(x) =

∞  (−1)n n=0

(n!)2

xn

which could also be cast in terms of the hypergeometric function 1 F 2 as u(x) = 1 − x (1 F 2 [1; 2, 2; −x]) For the second solution, we need σn (0), φn (0) = σn (0) = (2n) &

(−1)n 2n (n!)2

Because m = rb − ra = 0, we set b0 = 0, and the other coefficients are given by bn

= = =

−

Qn,n−1 (0)bn−1 + σn (0) Qn,n (0)

1 (−1)n (−1)n (−1)n (−1)n b − 2 = b − 2 + · · · − 2 n−1 0 n2 n(n!)2 (n!)2 (n!)2 n(n!)2   (−1)n 1 1 −2 1 + + ··· + (n!)2 2 n −

Thus the second solution is given by v(x) = u(x) ln(x) − 2

 n  1 (n!)2 k

∞  (−x)n n=1

k=1

and the complete solution is y = Au(x) + Bv(x).

EXAMPLE I.3.

Given the equation

d2 y dy + 3x + (2x − 8)y = 0 2 dx dx The terms for & ρi,j are & ρ2,0 = 9, & ρ1,0 = 3, & ρ0,0 = −8, and & ρ0,1 = 2. The indicial roots are ra = 4/3 and rb = −2/3. The difference is an integer; then m = 2. The only nonzero values of Qn,k are   Qn,n (r) = n 9n + 9(2r − 1) + 3 and Qn,n−1 (r) = 2 9x2

751

752

Appendix I: Additional Details and Fortification for Chapter 9

Thus & φ0 (r) = 1 and & φn (r) = −

2 & φn−1 (r) n (9n + 9(2r − 1) + 3)

Using the larger root, ra = 4/3,     4 −2 4 & & φn = φn−1 3 9n(n + 2) 3        −2 −2 −2 4 & = ··· φ0 9n(n + 2) 9(n − 1)(n + 1) 9·1·3 3 =

(−1)n 2n+1 9n (n!)(n + 2)!

The first solution is then given by u(x) =

∞  n=0

(−1)n 2n+1 xn+(4/3) 9n (n!)(n + 2)!

or in terms of hypergeometric functions,   ' ( 2x 2x u(x) = x4/3 1 − 1; 2, 4; − F 1 2 27 9 Because m = 2, we only need & φ1 (rb) for the second solution,   2 −2 2 & φ1 − = = 3 9(−1) 9 Next, we need σn (ra ) and η, σn (ra )

=

φn (ra ) = [9(2ra + 2n − 1) + 3] &

η

=

−

(−1)n 2n+1 18(n + 1) 9n (n!)(n + 2)!

φm−1 (rb) Qm,m−1& 2 =− 2 σ0 (ra ) 9

For the coefficients bn , we have b0 = 1, b1 = 2/9, b2 = 0 and the rest are found by recurrence, that is, bn

= = = =

Qn,n−1 (rb) σn−m (ra ) bn−1 − η Qn,n (rb) Qn,n (rb)    −2 (n − 1) (−1)n 2n+1 bn−1 + 9n(n − 2) 9n (n − 2)!n! n(n − 2)     (−2)n−2 2 2 (−1)n 2n+1 (n − 1) b2 + + ··· + 9n−2 n!(n − 2)! 9n (n − 2)!n! (3)(1) (n)(n − 2)  n−3  (−1)n 2n+1  (n − 1 − k) , for n > 2 9n (n − 2)!n! (n − k)(n − 2 − k) −

k=0

Appendix I: Additional Details and Fortification for Chapter 9

753

The second solution is then given by v(x)

2 2 x−2/3 + x1/3 − u ln(x) 9 81     ∞ n−3  (−1)n 2n+1 (n − 1 − k) n−(2/3) + x 9n (n − 2)!n! (n − k)(n − 2 − k)

=

n=3

k=0

and the complete solution is y = Au(x) + Bv(x).

I.4 Series Solution of Legendre Equations I.4.1 Legendre Equations The Legendre equation of order μ is given by the following equation:  d2 y  dy 1 − x2 − 2x + μ (μ + 1) y = 0 (I.20) dx2 dx Using the series solution expanded around the ordinary point x = 0, we seek a solution of the form ∞  y= an xn n=0

With N = 2, the coefficients ρi,j are: ρ2,0 = 1, ρ2,2 = −1, ρ1,1 = −2, and ρ0,0 = μ (μ + 1). Based on (9.21), the only nonzero values are for n = k, that is, φn,n

=

−

ρ0,0 + ρ1,1 (n) + ρ2,2 (n)(n − 1) μ(μ + 1) − n(n + 1) =− ρN,0 (n + 1)(n + 2) (n + 1)(n + 2)

=

−

(μ + n + 1)(μ − n) (n + 1)(n + 2)

which yields the following recurrence equation: an+2 = −

(μ + n + 1)(μ − n) an (n + 1)(n + 2)

When separated according to even or odd subscripts, with n ≥ 1, a2n

=

n−1 (−1)n  [μ − 2(n − k)] [μ + 2(n − k) + 1] a0 (2n)!

(I.21)

n−1 (−1)n  [μ − 2(n − k) − 1] [μ + 2(n − k) + 2] a1 (2n + 1)!

(I.22)

k=0

a2n+1

=

k=0

where a0 and a1 are arbitrary. Let functions 2n (μ) and 2n+1 (μ) be defined as 2n (μ)

=

n−1 (−1)n  (μ − 2(n − k)) (μ + 2(n − k) + 1) (2n)!

(I.23)

k=0

2n+1 (μ)

=

n−1 (−1)n  (μ − 2(n − k) − 1) (μ + 2(n − k) + 2) (I.24) (2n + 1)! k=0

754

Appendix I: Additional Details and Fortification for Chapter 9

then the solution to the Legendre equation of order μ is     ∞ ∞   y = a0 1 + 2n (μ)x2n + a1 x + 2n+1 (μ)x2n+1 n=1

(I.25)

n=1

The two infinite series are called the Legendre functions of the second kind, namely Leven (x) and Lodd (x), where =

Leven (x)

∞ 

1+

2n (μ)x2n

(I.26)

2n+1 (μ)x2n+1

(I.27)

n=1

=

Lodd (x)

∞ 

x+

n=1

For the special case when μ = μeven is an even integer, μeven +2j (μeven ) = 0, j = 1, . . ., and thus Leven (x) becomes a finite sum. Similarly, when μ = μodd is an odd integer, μodd +2j (μodd ) = 0, j = 1, . . ., and Lodd (x) becomes a finite sum. In either case, the finite sums will define a set of important polynomials. By carefully choosing the values of a0 and a1 , either of the finite polynomials can be normalized to be 1 at x=1. If μ = μeven = 2, we need a0 = A(−1)

(2)! 2 (!)2

(I.28)

Conversely, if μ = μodd = 2 + 1, a1 = A(−1)

(2 + 2)! 2 !( + 1)!

(I.29)

where A is arbitrary. Thus with these choices for a0 and a1 , we can rewrite (I.25) to be y = APn (x) + BQn (x)

(I.30)

where n is an integer. Qn is the Legendre function that is an infinite series, whereas Pn is a finite polynomial referred to as Legendre polynomial of order n and given by 

Int(n/2)

Pn (x) =

k=0

where

+ Int(n/2) =

(−1)k [2n − 2k]! xn−2k − k)!(n − 2k)!

2n k!(n

n/2 (n − 1)/2

if n even if n odd

(I.31)

(I.32)

The Legendre functions, Qn (x), has a closed form that can be obtained more conveniently by using the method of order reduction. Applying d’Alembert’s method of order reduction, we can set Qn (x) = q(x)Pn (x), where q(x) is obtained via Lemma I.1 given in Section I.2. Applying this approach to (I.20),     dz   2 2 dPn + 2 1−x 0 = 1 − x Pn − 2xPn z dx dx dz z

=

2x dx dPn −2 2 1−x Pn

Appendix I: Additional Details and Fortification for Chapter 9

z

= =

 Thus with q = − zdz,

 

1

exp

(Pn )2

2x 1 − x2

755

 dx

−1 (1 − x2 ) (Pn )2  8

Qn (x) = Pn (x)

1 (1 − x2 ) (Pn (x))2

9 dx

(I.33)

where we included a factor of (−1) to make it consistent with (I.26) and (I.27).

I.4.2 Associated Legendre Equation A generalization of the Legendre equation (I.20) is the associated Legendre equation given by     2 dy m2 2 d y 1−x − 2x + n(n + 1) − y=0 (I.34) dx2 dx 1 − x2 Note that if m = 0, we get back the Legendre equation. We now consider the situation in which n and m are nonnegative integers. Instead of solving (I.34) by series solution, we approach the solution by using a change of variable, namely let w = (1 − x2 )−m/2 y

(I.35)

With y = qw, where q = (1 − x2 )m/2 , the terms on the right-hand side of (I.34) can each be divided by q and then evaluated to be     1 m2 m2 n(n + 1) − y = n(n + 1) − w q 1 − x2 1 − x2 2x dy q dx

=

1 − x2 d2 y q dx2

=

−

2mx2 dw w − 2x 1 − x2 dx    2 m (m − 1)x2 − 1 dw  2 d w w − 2mx + 1 − x 1 − x2 dx dx2

Doing so reduces (I.34) to   d2 w dw 1 − x2 − 2(m + 1)x + (n − m)(n + m + 1)w = 0 dx2 dx

(I.36)

Now let S be defined by S(x) = APn (x) + BQn (x)

(I.37)

Then S satisfies the Legendre equation given by (I.20). With f (x) = 1 − x2 , df/dx = −2x and a = n(n + 1), (I.20) can be rewritten with S replacing y, as f

d2 S df dS + + aS = 0 dx2 dx dx

(I.38)

756

Appendix I: Additional Details and Fortification for Chapter 9

Furthermore, with d2 f/dx2 = −2 and dk f/dxk = 0 for k > 2, the mth derivative of each term in (I.38) is, using the Leibnitz rule (9.6), dm dxm



d2 S f 2 dx

 =

=

dm dxm



df dS dx dx

 =

= dm (aS) dxm

=

  k   (2+m−k)  m   d f d S m k dxk dx(2+m−k) k=0      (1+m)  d(2+m) S df S d f +m (2+m) (1+m) dx dx dx   m  m(m − 1) d2 f d S + 2 2 dx dxm     k+1  m   d f d(1+m−k) S m k dxk+1 dx(1+m−k) k=0    2  m  df d(1+m) S d f d S + m dx dx(1+m) dx2 dxm dm S dxm

a

and adding all the terms together, we obtain   d2 1 − x2 dx2



dm S dxm

 − 2(m + 1)x

d dx



dm S dxm



 + (n − m)(n + m + 1)

dm S dxm

 =0 (I.39)

Comparing (I.39) with (I.36), w

=

 −m/2 1 − x2 y

=

dm S dxm dm Pn dm Qn A m +B dx dxm

Thus the solution to the associated Legendre equation (I.34) is y= APn,m (x) +  BQn,m (x)

(I.40)

where Pn,m and Qn,m are the associated Legendre polynomials and associated Legendre functions, respectively, of order n and degree m defined by2

2

Pn,m

=

Qn,m

=

 m/2 dm (−1)m 1 − x2 Pn (x) dxm  m/2 dm (−1)m 1 − x2 Qn (x) dxm

(I.41)

In some references, the factor (−1)m is neglected, but we chose to include it here because MATLAB happens to use the definition given in (I.41).

Appendix I: Additional Details and Fortification for Chapter 9

757

I.5 Series Solution of Bessel Equations I.5.1 The Bessel equation of order ν is given by the following differential equation: x2

 d2 y dy  2 +x + x − ν2 y = 0 2 dx dx

(I.42)

Using a series expansion around the regular singular point x = 0, we can identify the ρ1,0 = 1, & ρ0,0 = −ν2 , and & ρ0,2 = 1. The indicial roots following coefficients: & ρ2,0 = 1, & using (9.2) are ra = ν and rb = −ν. Applying the Frobenius method summarized in Theorem 9.2, the only nonzero values of Qn,k are Qn,n−2 (r) = 1

Qn,n (r) = n(n + 2r)

and

φ1 (r) = 0, & φn (r) = −& φn−2 (r)/[n(n + 2r)], for n > 1, and σn (r) = (2r + thus & φ0 (r) = 1, & & φ1 (r) = 0, functions corresponding to odd subscripts 2n)φn (r). Furthermore, because & will be zero, that is, & φ2n+1 (r) = 0

for n = 0, 1, . . .

For those with even subscripts, & φ2n (r)

= =

−1 & φ2n−2 = 4n(n + r)



   −1 −1 & ··· φ0 4n(n + r) 4(1)(1 + r)

(−1)n

4n n!

n−1 k=0

(n + r − k)

Depending on the value of the order ν, we have the various cases to consider: r Case 1: 2ν is not an integer. We have a 2k+1 = b2k+1 = 0, k = 0, 1, . . ., and for n = 1, 2, . . . a2n =

(−1)n  n−1 4n n! k=0 (n + ν − k)

and

b2n =

(−1)n  n−1 4n n! k=0 (n − ν − k)

The two independent solutions are then given by u(x) =

∞  n=0

(−1)n x2n+ν  4n n! n−1 k=0 (n + ν − k)

and

v(x) =

∞  n=0

(−1)n x2n−ν  4n n! n−1 k=0 (n − ν − k)

These results can further be put in terms of Gamma functions (cf. (9.9)), and after extracting constants out of the summations, we obtain u(x) = 2ν (ν + 1)J ν (x)

and

v(x) = 2−ν (−ν + 1)J −ν (x)

where J ν (x) is known as the Bessel function of the first kind defined by J ν (x) =

∞    x 2n+ν n=0

2

(−1)n n!(n + ν + 1)

(I.43)

where the order ν in the definition (I.43) may or may not be an integer. Thus in terms of Bessel functions, the complete solution, for ν not an integer, is given by y = AJ ν (x) + BJ −ν (x)

(I.44)

758

Appendix I: Additional Details and Fortification for Chapter 9

r Case 2: 2ν is an odd integer. Let m = r − r = 2ν be an odd integer . Because, a b & φk = 0 when k is odd, the value η in (9.43) will be zero. This means that b2n = & φ2n (−ν), and we end up with the same result as that of case 1, that is, y = AJ ν (x) + BJ −ν (x)

(I.45)

r Case 3: 2ν = 0 is an even integer. Let ν =  with  an integer. For the first root ra = , we have a2n = & φ2n () and the first solution becomes u(x) = 2 !J  (x)

(I.46)

For the second solution, we will separate v(x) into three parts: v1 , v2 , and v3 , where v1 contains the terms with b2n (x), n < , v2 is the term with ln(x) and v3 contains the rest of the terms. ( − n − 1)! φ2n (−) = n For v1 , we take n < , for which b2n = & and obtain 4 n!( − 1)! v1 (x) =

−1 2n−  x ( − n − 1)!

4n n!( − 1)!

n=0

' =

( −1   1 x 2n− ( − n − 1)! (I.47)  2 ( − 1)! 2 n! n=0

For v2 , with m = 2, we find that η=−

φ2−2 (−) Q2,2−2 (−)& 2 =−  σ0 () 4 !( − 1)!

and together with u(x) in (I.46), we obtain '

( 1 J  (x) ln(x) v2 (x) = ηu(x) ln(x) = −2  2 ( − 1)!

(I.48)

For v3 , one can first show that b2(n+)

= =

−

Q2(n+),2(n−1+) (−)b2(n−1+) + ησ2n () Q2(n+),2(n+) (−)

(−1)n b2 4n n!(n + )! (' ' ( 8 9 n−1  (−1)n 1 1 1 +  + 4 ( − 1)! 4n n!(n + )! n−k n−k+ k=0

Because bm = b2 = 0, we obtain v3 (x) to be 8 ( 9 ∞ n   x 2n+ (−1)n  1 1 1 v3 (x) =  + (I.49) 2 ( − 1)! 2 n!(n + )! k k+

'

n=1

k=1

Appendix I: Additional Details and Fortification for Chapter 9

759

Adding up (I.47), (I.48) and (I.49), we have the second solution v(x) as v(x)

= =

v1 (x) + v2 (x) + v3 (x) ' (5 −1    1 x 2n− ( − n − 1)! −  2J  (x) ln(x) − 2 ( − 1)! 2 n! n=0  n  G ∞     1 x 2n+ (−1)n 1 − + (I.50) 2 n!(n + )! k k+ n=1

k=1

A more standard solution formulation known as the Weber form is given by y = AJ  (x) + BY (x)

(I.51)

where the function Y (x) is known as Bessel function of the second kind (also known as the Neumann function), defined as  x  1   x 2n− ( − n − 1)! 2 J  (x) ln +γ − π 2 π 2 n! n=0 8 n+ 9 ∞ 1 1   x 2n+ (−1)n − π 2 n!(n + )! k −1

Y (x)

=

n=0

(I.52)

k=1

where γ is known as Euler’s constant, defined by '   ( 1 1 1 + + ··· + γ = lim − ln(n) = 0.572215664 . . . n→∞ 2 n

(I.53)

r Case 4: ν = 0. With η = 1, a similar procedure as in Case 3 above will lead to a solution of the same Weber form, y = AJ 0 (x) + BY0 (x)

(I.54)

where  x  2   x 2n (−1)n 2 Y0 (x) = J 0 (x) ln +γ − π 2 π 2 (n!)2 ∞

n=1

 n  1 k

(I.55)

k=1

An alternative method for computing the Bessel functions is to define the Bessel function of the second kind as Yν (x) =

J ν (x) cos(νπ) − J −ν (x) sin(νπ)

(I.56)

Then for ν = n, an integer, we simply take the limit, that is, Yn (x) = lim Yν (x) ν→n

(I.57)

This means we can unify the solutions to both cases of ν being an integer or not, as y(x) = AJ ν (x) + BYν (x)

(I.58)

760

Appendix I: Additional Details and Fortification for Chapter 9

I.5.2 Bessel Equations of Parameter λ A simple extension to the Bessel equation is to introduce a parameter λ in the Bessel equation as follows x2

 d2 y dy  2 2 +x + λ x − ν2 y = 0 dx2 dx

(I.59)

Instead of approaching the equation directly with a series solution, we could simply use a change of variable, namely w = λx. Then dx =

1 dw λ

;

dy dy =λ dx dw

;

2 d2 y 2d y = λ dx2 dw2

Substituting these into (I.59), we get w2

 d2 y dy  2 +w + w − ν2 y = 0 dw2 dw

whose solution is given by y = AJ ν (w) + BYν (w) or y = AJ ν (λx) + BYν (λx)

(I.60)

I.5.3 Modified Bessel Equations and Functions The modified Bessel equations of order ν is given by  d2 y dy  2 +x − x + ν2 y = 0 2 dx dx √ which is just the Bessel equation with parameter i = −1, that is,

x2

x2

(I.61)

 d2 y dy  2 2 +x + (i) x − ν2 y = 0 2 dx dx

Then the solution is given by y = AJ ν (ix) + BYν (ix) Another form of the solution is given by y = AIν (ix) + BKν (ix)

(I.62)

where Iν (x) is the modified Bessel equation of the first kind of order ν defined by   νπi J ν (ix) Iν (x) = exp − (I.63) 2 and Kν (x) is the modified Bessel equation of the second kind of order ν defined by   (ν + 1)πi Kν (x) = exp (I.64) [J ν (ix) + iYν (ix)] 2

Appendix I: Additional Details and Fortification for Chapter 9

761

I.6 Proofs for Lemmas and Theorems in Chapter 9 I.6.1 Proof of Series Expansion Formula, Theorem 9.1 Assuming a series solution of the form y=

∞ 

an xn

(I.65)

n=0

the derivatives are given by dy dx d2 y dx2

=

∞ 

nan xn−1 =

n=1

=

∞  (n + 1)an+1 xn n=0

∞ 

(n + 1)(n)an+1 x

n−1

∞  = (n + 2)(n + 1)an+2 xn

n=1

n=0

.. . dN y dxN

=

∞  (n + N)!

n!

n=0

an+N xn

(I.66)

After substitution of (9.18) and (I.66) into (9.17), while using (9.5),  ∞ n  N    (k + j )! n ak+j ρ j,n−k x =0 k! k=0 j =0

n=0

Because x is not identically zero, we have  n  N   (k + j )! ak+j ρ j,n−k =0 k!

for n = 0, 1, . . . , ∞

(I.67)

k=0 j =0

For a fixed n, let μ j,k

(k + j )! = ρ j,n−k k!

→

μ j,m−j =

⎧ ⎨ ρ0,n−m ⎩

if j = 0

ρ j,n−m+j

 j −1 i=0

(m − i)

if j > 0

We can rearrange the summation in (I.67) to have the following structure: j =0 μ0,0 μ0,1

j =1

μ0,2 .. .

μ1,1

..

.

μ1,2 .. .

..

.

μN,0

..

.

..

.

μN,1 μN,2 .. .

μ0,n

···

j =N a0 a1

μ1,0

μ1,n

μN,n

.. .

an+N

where the group of terms to the left of am are summed up as the coefficient of am . Note that μ j, = 0 if  < 0. In addition, we can define ρ j, = 0 for  < 0, and

762

Appendix I: Additional Details and Fortification for Chapter 9

obtain μ j,m−j = 0 for m − j > n. Thus the coefficients of am for m ≤ (n + N) can be formulated as ⎧ N  ⎪ ⎪ ⎪ μ j,m−j if m < n + N ⎨ μ0,m + j =1 coef (am ) = ⎪ ⎪ ⎪ ⎩ μ if m = n + N N,n

Letting a0 , a1 , . . . , aN−1 be arbitrary, we have for n = 0, 1, . . ., ⎛ ⎞ n+N−1 N   μN,n an+N + am ⎝μ0,m + μ j,m−j ⎠ = 0 j =1

m=0

n+N−1 

an+N

= =

−

⎛ am ⎝μ0,m +

⎞ μ j,m−j ⎠

j =1

m=0

n+N−1 

N 

μN,n φn,m am

m=0

where ρ0,n−m +

N 

ρ j,n−m+j

j =1

φn,m = (−1)

ρN,0

j −1  (m − i) i=0

N 

(n + i)

i=1

and ρ j, = 0

 0. Then v, x(dv/dx) and x2 (d2 v/dx2 ) becomes v

∞ 

=

ηu ln(x) +

=

' (  ∞ du η u + x ln(x) + bn (n + rb)xn+rb dx

=

' (  ∞ du d2 u 2 + x ln(x) 2 + η −u + 2x bn (n + rb)(n + rb − 1)xn+rb dx dx

bn xn+rb

n=0

x

x2

dv dx

d2 v dx2

n=0

n=0

Appendix I: Additional Details and Fortification for Chapter 9

Substituting into P2 (x) x2& we have

d2 v dv & + x& P1 (x) + P0 (x)v = 0 2 dx dx 

du & η ln(x) + x& P1 (x) + P0 (x)u dx  '  ( du +η & P2 (x) −u + 2x +& P1 (x)u dx

=

0



d2 u P2 (x) 2 x2& dx

+& P2 (x)

∞ 

bn (n + rb)(n + rb − 1)xn+rb

n=0

+& P1 (x)

∞ 

bn (n + rb)xn+rb

n=0

+& P0 (x)

∞ 

bn xn+rb

n=0

Because u is a solution to the differential equation, the group of terms multiply ρi,n xn and u(x) = ing ln(x) is equal to zero. After substitution of & Pi (x) = ∞ n=0 & ∞ n+ra & , the equation above becomes n=0 φn (ra )x ∞ 

n 

η

n+ra

x

n=0

& φk (ra ) (& ρ1,n−k + (2ra + 2k − 1)& ρ2,n−k )

k=0

+

∞ 

xn+rb

n 

n=0

bk Qn,k (rb)

=

0

k=0

With ra = rb + m, the first summation can be reindexed, that is, ∞ 

xn+rb η

n=m

n−m 

& φk (ra ) (& ρ1,n−m−k + (2ra + 2k − 1)& ρ2,n−m−k )

k=0

+

∞ 

xn+rb

n=0

n 

bk Qn,k (rb)

=

0

k=0

Using the definition of σn (r) given in (9.40), we arrive at the working equation,  m−1 n   n+rb x bk Qn,k (rb) n=0

 rb +m

+x

ησ0 (ra ) + bm Qm,m (rb) +

 +

k=0

∞ 

n=m+1

8 n+rb

x

ησn−m (ra ) +

m−1 

 bk Qm,k (rb)

k=0 n  k=0

bk Qn,k (rb)

9 =

0

763

764

Appendix I: Additional Details and Fortification for Chapter 9

Thus for n < m, the formula for bn becomes those for & φn (rb). For n = m, note that Qm,m (rb) = 0, and we have bm arbitrary, which we can set to zero. Doing so and making the coefficient of xm+rb be equal to zero, m−1 η=−

k=0

bk Qm,k (rb) σ0 (ra )

For n > m > 0, each coefficient of xn+rb can be set to zero, which yields the recurrence formula for bn ,  ησn−m (ra ) + n−1 k=0 Qn,k (rb)bk bn = − Qn,n (rb) Finally, if m = 0, a similar derivation can be followed, except that we can set η = 1 as discussed before. The working equation is now given by xrb (σ0 (ra ) + bm Qm,m (rb)) 9 ∞ 8 n   n+rb x bk Qn,k (rb) + σn−m (ra ) + n=1

=

0

k=0

ρ1,0 /& ρ2,0 )/2, which means σ0 = 0. With Note that for this case, ra = rb = (1 − & Q0,0 (rb) = 0, b0 can be arbitrary and thus can be set to be zero. The remaining coefficients then become  σn (ra ) + n−1 k=0 Qn,k (rb)bk bn = − Qn,n (rb)

I.6.3 Proof of Bessel Function Identities 1. Derivatives of J ν (x). Recall the definition of J ν (x), J ν (x) =

∞  m=0

 x 2m+ν (−1)m m!(m + ν + 1) 2

To show (9.63), multiply J ν (x) by xν and then take the derivative with respect to x, 8∞ 9 d ν d  (−1)m x2m+2ν (x J ν (x)) = dx dx m!(m + ν + 1)22m+ν m=0

=

∞  (−1)m (2m + 2ν)x2m+2ν−1 m!(m + ν + 1)22m+ν

m=0

=

xν

∞  m=0

=

 x 2m+ν−1 (−1)m m!(m + ν) 2

xν J ν−1 (x)

Appendix I: Additional Details and Fortification for Chapter 9

To show (9.64), multiply J ν (x) by x−ν and then take the derivative with respect to x, 8∞ 9  d  −ν d  (−1)m x2m x J ν (x) = dx dx m!(m + ν + 1)22m+ν m=0

=

∞  m=1

=

∞  m=1

(−1)m (2m)x2m−1 m!(m + ν + 1)22m+ν (−1)m x2m−1 (m − 1)!(m + ν + 1)22m+ν−1

=

−x−ν

=

−ν

∞  m=0

 x 2m+ν+1 (−1)m m!(m + ν + 2) 2

−x J ν+1 (x)

To show (9.65), expand the derivative operation on xν J ν (x) d ν d (x J ν (x)) = νxν−1 J ν (x) + xν J ν (x) dx dx and equate with (9.63) to obtain νxν−1 J ν (x) + xν

d J ν (x) dx d J ν (x) dx

=

xν J ν−1 (x)

=

ν J ν−1 (x) − J ν (x) x

To show (9.66), expand the derivative operation on x−ν J ν (x)  d  −ν d x J ν (x) = −νx−ν−1 J ν (x) + x−ν J ν (x) dx dx and equate with (9.64) to obtain −νx−ν−1 J ν (x) + x−ν

d J ν (x) dx d J ν (x) dx

=

−x−ν J ν+1 (x)

=

ν −J ν+1 (x) + J ν (x) x

2. Derivatives of Yν (x). Recall the definition of Yν (x), Yν (x)

=

ν−1  2  x 1  (ν − m − 1)!  x 2m−ν ln + γ J ν (x) − π 2 π m! 2 m=0 8 9 ∞ m 1  (−1)m  x 2m+ν  1 − π m!(m + ν)! 2 k m=1 k=1 8m+ν 9 ∞ 1  (−1)m  x 2m+ν  1 − π m!(m + ν)! 2 k m=0

k=1

765

766

Appendix I: Additional Details and Fortification for Chapter 9

To show (9.67), multiply Yν (x) by xν and then take the derivative with respect to x, while incorporating (9.63),

d ν (x Yν (x)) dx

=

  2 d   x  ln + γ xν J ν (x) π dx 2 −

ν−1 1 d  (ν − m − 1)!x2m π dx m!22m−ν m=0

8 m 9 ∞ 1 1 d  (−1)m x2m+2ν − π dx m!(m + ν)!22m+ν k m=1

k=1

8m+ν 9 ∞ 1 1 d  (−1)m x2m+2ν − π dx m!(m + ν)!22m+ν k m=0

=

k=1

 2 ν−1 2  x x J ν (x) + ln + γ xν J ν−1 (x) π π 2 −

ν−1 1  (ν − m − 1)!x2m−1 π (m − 1)!22m−ν−1 m=1

8 m 9 ∞ 1 1 (−1)m x2m+2ν−1 − 2m+ν−1 π m!(m + ν − 1)!2 k m=1

k=1

∞ 1 (−1)m x2m+2ν−1 − π m!(m + ν − 1)!22m+ν−1 m=0

8m+ν−1 9  1 k k=1

2 ν−1  (−1)m  x 2m+ν x π m!(m + ν)! 2 ∞

−

m=0

=

 2  x ln + γ xν J ν−1 (x) π 2 1 ν  (ν − m)!  x 2m−ν+1 x π (m)! 2 ν

−

m=0

8 m 9 ∞  x 2m+ν−1  1 ν (−1)m 1 − x π m!(m + ν − 1)! 2 k m=1

 x 2m+ν−1 (−1)m 1  − xν π m!(m + ν − 1)! 2 ∞

m=0

=

xν Yν−1 (x)

k=1

8m+ν−1 9  1 k k=1

Appendix I: Additional Details and Fortification for Chapter 9

To show (9.68), multiply Yν (x) by x−ν and then take the derivative with respect to x, while incorporating (9.64),  d  −ν x Yν (x) dx

=

  2 d   x  ln + γ x−ν J ν (x) π dx 2 ν−1 1 d  (ν − m − 1)!x2m−2ν π dx m!22m−ν m=0 8 m 9 ∞ 1 1 d  (−1)m x2m − π dx m!(m + ν)!22m+ν k m=1 k=1 8 9 ∞ m+ν 1 1 d  (−1)m x2m − π dx m!(m + ν)!22m+ν k

−

m=0

=

k=1

 2 −ν−1 2  x x J ν (x) − ln + γ x−ν J ν+1 (x) π π 2 +

ν−1 1  (ν − m)!x2m−2ν−1 π m!22m−ν−1 m=0

8 m 9 ∞ 1 1 (−1)m x2m−1 − 2m+ν−1 π (m − 1)!(m + ν)!2 k m=1 k=1 8m+ν 9 ∞ 1 1 (−1)m x2m−1 − 2m+ν−1 π (m − 1)!(m + ν)!2 k m=1

=

k=1

 2  x − ln + γ x−ν J ν+1 (x) π 2 ν  1 (ν − m)!  x 2m−ν−1 + x−ν π (m)! 2 m=0

8 m 9 ∞  x 2m+ν+1  1 −ν  (−1)m 1 + x π m!(m + ν + 1)! 2 k m=1

+ =

1 −ν x π

∞  m=0

k=1

8 9  x 2m+ν+1 m+ν+1  1 (−1)m m!(m + ν + 1)! 2 k k=1

−ν

−x Yν−1 (x)

To show (9.69), expand the derivative operation on xν Yν (x) d ν d (x Yν (x)) = νxν−1 Yν (x) + xν Yν (x) dx dx and equate with (9.67) to obtain νxν−1 Yν (x) + xν

d Yν (x) dx d Yν (x) dx

=

xν Yν−1 (x)

=

ν Yν−1 (x) − Yν (x) x

767

768

Appendix I: Additional Details and Fortification for Chapter 9

To show (9.70), expand the derivative operation on x−ν Yν (x)  d  −ν d x Yν (x) = −νx−ν−1 Yν (x) + x−ν Yν (x) dx dx and equate with (9.68) to obtain −νx−ν−1 Yν (x) + x−ν

d Yν (x) dx d Yν (x) dx

=

−x−ν Yν+1 (x)

=

ν −Yν+1 (x) + Yν (x) x

3. Derivatives of Iν (x). Recall the definition of Iν (x),  νπ  Iν (x) = exp − i J ν (ix) 2 To show (9.71), multiply Iν (x) by xν and then take the derivative with respect to x, while using (9.65), d ν x Iν (x) dx

= = =

 νπ     ν exp − i νxν−1 J ν (ix) + xν iJ ν−1 (ix) − J ν (ix) 2 x   (ν − 1)π xν exp − i J ν−1 (ix) 2 xν Iν−1 (x)

To show (9.72), multiply Iν (x) by xν and then take the derivative with respect to x, while using (9.66), d −ν x Iν (x) dx

= = =

 νπ     ν exp − i −νx−ν−1 J ν (ix) + xν −iJ ν+1 (ix) + J ν (ix) 2 x   (ν + 1)π i J ν+1 (ix) x−ν exp − 2 x−ν Iν+1 (x)

To show (9.73), expand the derivative operation on xν Iν (x) d ν d (x Iν (x)) = νxν−1 Iν (x) + xν Iν (x) dx dx and equate with (9.71) to obtain νxν−1 Iν (x) + xν

d Iν (x) dx d Iν (x) dx

=

xν Iν−1 (x)

=

ν Iν−1 (x) − Iν (x) x

To show (9.74), expand the derivative operation on x−ν Iν (x)  d d  −ν x Iν (x) = −νx−ν−1 Iν (x) + x−ν Iν (x) dx dx

Appendix I: Additional Details and Fortification for Chapter 9

and equate with (9.72) to obtain −νx−ν−1 Iν (x) + x−ν

d Iν (x) dx d Iν (x) dx

=

x−ν Iν+1 (x)

=

ν Iν+1 (x) + Iν (x) x

4. Derivatives of Kν (x). Recall the definition of Iν (x),  Kν (x) = exp

 (ν + 1)π i (J ν (ix) + iYν (ix)) 2

To show (9.75), multiply Kν (x) by xν and then take the derivative with respect to x, while using (9.65) and (9.69), d ν x Kν (x) dx



=

= =

 (ν + 1)π  ν−1 exp i νx (J ν (ix) + iYν (ix)) 2   ν ν + xν iJ ν−1 (ix) − J ν (ix) − Yν−1 (ix) − i Yν (ix) x x   (ν)π − xν exp i (J ν−1 (ix) + iYν−1 (ix)) 2 xν Kν−1 (x)

To show (9.72), multiply Iν (x) by xν and then take the derivative with respect to x, while using (9.66) and (9.70), d −ν x Kν (x) dx



=

= =

 (ν + 1)π  i −νx−ν−1 (J ν (ix) + iYν (ix)) 2   ν ν + x−ν −iJ ν+1 (ix) + J ν (ix) + Yν+1 (ix) + i Yν (ix) x x   (ν + 2)π i (J ν+1 (ix) + iYν+1 (ix)) − x−ν exp 2

exp

− x−ν Kν+1 (x)

To show (9.77), expand the derivative operation on xν Kν (x) d ν d (x Kν (x)) = νxν−1 Kν (x) + xν Kν (x) dx dx and equate with (9.75) to obtain νxν−1 Kν (x) + xν

d Kν (x) dx d Kν (x) dx

=

−xν Kν−1 (x)

=

ν −Kν−1 (x) − Iν (x) x

To show (9.78), expand the derivative operation on x−ν Kν (x)  d  −ν d x Kν (x) = −νx−ν−1 Kν (x) + x−ν Kν (x) dx dx

769

770

Appendix I: Additional Details and Fortification for Chapter 9

and equate with (9.72) to obtain d Kν (x) = −x−ν Kν+1 (x) dx d ν Kν (x) = −Kν+1 (x) + Kν (x) dx x 5. Bessel functions of negative integral orders. We use induction to prove the identity. The recurrence formula yields the following two relationships, −νx−ν−1 Kν (x) + x−ν

2n J −n (x) − J −n+1 (x) x 2n J n−1 (x) = J n (x) − J n+1 (x) x Adding and subtracting these equations, J −n−1 (x)

J −n−1 (x)

J −n−1 (x)

=

=

=

−

2n (J −n (x) − J n (x)) x − (J −n+1 (x) + J n−1 (x)) − J n+1 (x)

(I.68)

2n (J −n (x) + J n (x)) x − (J −n+1 (x) − J n−1 (x)) + J n+1 (x)

(I.69)

−

−

If n is even, while using the inductive hypothesis, that is, supposing that J n (x) = J −n (x) and J n−1 (x) = −J −n+1 (x), we can then use (I.68) and see that J −(n+1) (x) = −J n+1 (x) If n is odd, while using the inductive hypothesis, that is, supposing that J n (x) = −J −n (x) and J n−1 (x) = J −n+1 (x), we can then use (I.69) and see that J −(n+1) (x) = J n+1 (x) To complete the proof, we note that J 0 (x) = (−1)0 J 0 (x) and with the recurrence formula, J −1 (x) = −J 1 (x) We can then continue the induction process to show that the identity is satisfied for n = 2, 3, . . . and conclude that J −n (x) = (−1)n J n (x) Similar approaches can be used to show the identities for Y−n (x), I−n (x) and K−n (x).

APPENDIX J

Additional Details and Fortification for Chapter 10

J.1 Shocks and Rarefaction For the general quasilinear first-order PDEs, it is possible that the solutions of the characteristic equations will yield a surface that contains folds – resulting in multiple values of u for each point in some region of the space of independent variables. When this occurs, the classic solution (i.e., completely smooth solution) is not possible. Instead, a discontinuous solution that splits the domain into two or more regions with continuous surface solutions will have to suffice. A solution that covers both the classic solution and solutions with discontinuities are called weak solutions or generalized solutions. The discontinuities are known as shocks, and their paths can be traced as curves in the domain of the independent variables known as shock paths. We limit our discussion to PDEs whose independent variables are time 0 ≤ t < ∞ and a space dimension −∞ < x < ∞, given by the form ∂u ∂u + b(x, t, u) = c(x, t, u) ∂t ∂x

(J.1)

subject to a Cauchy condition u(x, t = 0) = u0 (x)

(J.2)

The method of characteristics immediately yields the following characteristic equations dt =1 ; ds

dx = b(x, t, u) ds

;

du = c(x, t, u) ds

(J.3)

subject to initial conditions, t(a, s = 0) = 0, x(a, s = 0) = a, u(a, s = 0) = u0 (a). The solution for t is immediately given by t = s. This reduces the problem to dx = b(x, s, u) ds

;

du = c(x, s, u) ds

(J.4)

which can be solved either analytically or numerically for fixed values of a, where a is the parameter along the Cauchy condition. Because of the coupling of the equations in (J.4), the solution for x and u is a curve C(x, u) that is parameterized by a and s. Unfortunately, these curves can contain folds, that is, several u values may correspond to a point (x, t). 771

772

Appendix J: Additional Details and Fortification for Chapter 10

To illustrate, consider the inviscid Burger equation given by ∂u ∂u +u =0 ∂t ∂x

(J.5)

with the Cauchy initial condition (J.2). Then the solution of (J.4) with b(x, s, u) = u, c(x, s, u) = 0, u(a, s = 0) = u0 (a), and x(a, s = 0) = a, is given by u(a, s) = u0 (a)

and

x(a, s) = u0 (a)s + a

Furthermore, let u0 (x) be given by ' ( 3 1 1 1 u0 (x) = − + 2 1 + eq(x) 2.5 + q(x) 2

 with

q(x) =

x − 10 10

2 (J.6)

We can plot u(a, s) versus x(a, s) at different fixed values of s with −80 ≤ a ≤ 100 as shown in Figure J.1. From the plots in Figure J.1, we see that as s increases, the initial shape moves to the right and slants more and more to the right. At s = 29.1, portions of the curve near x = 41.0 will have a vertical slope, and a fold is starting to form. When s = 80, three values of u correspond to values in the neighborhood of x = 78. At s = 120, portions of the curve near x = 54.8 will again have a vertical slope. Then at s = 300, we see that around x = 165 and x = 235, three values of u correspond to each of these x values. Finally, we see that at s = 600, there are five values of u that correspond to x = 370.

J.1.1 Break Times We refer to the values of s(= t) in which portions of the curves just begin to fold as the break times, denoted by sbreak . From the plots given in Figure J.1, we see that several shocks are possible, each with their respective break times. Assuming that the initial data u0 (a) are continuous, the shock that starts to form at the break time is along a characteristic that starts at a, which intersects with a neighboring characteristic that starts at a + . This means ∂x =0 ∂a

at s = sbreak

(J.7)

Suppose the shock at the break time will belong to a characteristic starting from a that belongs to a range [aleft , aright ]. For instance, one could plot the characteristics based on a uniform distribution of a and then determine adjacent values of a whose characteristics intersect, as shown in Figure J.2. The values of aleft and aright can then be chosen to cover this pair of adjacent values of a. The break time sbreak and the critical point acritical can then be determined by solving the following minimization problem   ∂x such that ≤0 (J.8) min {s} a∈[aleft , aright ] ∂a The values of x at sbreak along the characteristic corresponding to acritical will be the break position, denoted by xbreak , xbreak = x (acritical , sbreak )

(J.9)

Appendix J: Additional Details and Fortification for Chapter 10

s=0.0

773

s=29.1

1

1

0.9

0.9

0.8

0.8

u(a,s)

u(a,s)

x=41.0

0.7

0.6

0.5

0.7

0.6

−100

−50

0

50

100

0.5

150

−100

−50

0

x(a,s)

x=54.8

0.9

0.8

0.8

u(a,s)

u(a,s)

0.9

0.7

0.6

0.7

0.6

−50

0

50

100

150

0.5 −50

200

x(a,s)

0

50

200

s=600.0

0.9

0.9

0.8

0.8

u(a,s)

u(a,s)

150

1

x=235

x=165

100

x(a,s)

s=300.0

0.7

0.6

0.5 100

150

1

x=78.0

1

100

s=120.0

s=80.0 1

0.5

50

x(a,s)

0.7

0.6

150

200

250

300

350

0.5 300

400

x(a,s)

500

600

x(a,s)

Figure J.1. Plots of u versus x for different values of s, with −80 ≤ a ≤ 100.

t Figure J.2. Determination of aleft and aright .

x

t(=s)=0

a left

a right

774

Appendix J: Additional Details and Fortification for Chapter 10

150

100

Figure J.3. The characteristics corresponding to uniformly distributed values of a. Also included are two characteristics along acritical . The circles are the break points (xbreak , sbreak ).

t 50

0

−50

0

50

100

150

200

x

In particular, the characteristics (x, t) for the inviscid Burger equation (J.5) are given by straight lines t=

x−a u0 (a)

if

u0 (a) = 0

(J.10)

(If u0 (a) = 0, the characteristics are vertical lines at a.) For the initial data u0 (x) of (J.6), a set of characteristics corresponding to a set of uniformly distributed a values is shown in Figure J.3. From this figure, we could set [aleft , aright ] = [0, 50] to determine the break time of the first shock point. We could also set [aleft , aright ] = [−50, 0] to determine the break time of the other shock point. Solving the minimization problem of (J.8) for each of these intervals yields the following results: sbreak,1 = 29.1 sbreak,2 = 120

; ;

acritical,1 = 19.84 acritical,2 = −15.25

; ;

xbreak,1 = 41.0 xbreak,2 = 54.8

In Figure J.3, this information is indicated by two darker lines starting at (t, x) = (0, acritical ) and ending at the points (t, x) = (sbreak , xbreak ). These break times and break positions are also shown in Figure J.1 for s = 29.1 and s = 120 to be the correct values where portions of the curves are starting to fold.

J.1.2 Weak Solutions Once the break times and positions have been determined, a discontinuity in solution will commence as t = s increases and a weak solution has to be used. A function u(x, ˜ t) is a weak solution of a partial differential equation, such as (J.1), ∂u ∂u + b(x, t, u) = c(x, t, u) ∂t ∂x if

 0

∞



 ' ( ∂ u˜ ∂ u˜ ϑ(x, t) + b(x, t, u) − c(x, t, u) ˜ dx dt = 0 ∂t ∂x −∞ ∞

(J.11)

for all smooth functions of   ϑ (x, t), which has the property that ϑ = 0 for x outside some closed interval xleft , xright and for t outside of some closed interval tlow , thigh (with −∞ < xleft < ∞ and 0 ≤ tlow < thigh < ∞). The main idea of (J.11) is that via integration by parts, partial derivatives of discontinuous u(x, ˜ t) can be avoided by transferring the derivative operations instead on continuous functions ϑ(x, t).

Appendix J: Additional Details and Fortification for Chapter 10

775

u Area1 Figure J.4. The location of xshock based on equal area rule.

Area1 = Area2 Area2

xshock

x

Another important point is that the function ϑ(x, t) is kept arbitrary; that is, there is no need to specify this function nor the domain given by xright , xleft , tlow , or thigh . This will keep the number of discontinuities to a minimum. For instance, if a continuous u˜ can satisfy (J.11) for arbitrary ϑ, then no discontinuity need to be introduced, and u˜ = u, a classic solution. For the special case in which c(x, t, u) ˜ = c(x, t) is continuous, let the desired discontinuity that satisfies (J.11) occur at (t = s, xshock (s)). The value of xshock will occur when two characteristics, one initiated at a = a(−) and another initiated at a = a(+) , intersected to yield xshock . The condition (J.11) implies that xshock is located at a position where the area of the chopped region to right of xshock is equal to the area of the chopped region to the left of xshock , as shown in Figure J.4.

J.1.3 Shock Fitting Based on the equal area rule, a shock path xshock (s) with s ≥ sbreak can be determined by solving the following integral: 

a(+)

a(−)

'

∂x u(a, s) ∂a

( da = 0

(J.12)

    such that x a(−) , s = x a(+) , s = xshock (s). Generally, the location of the shock path, especially one that is based on the equal area rule, will require numerical solutions. We outline a scheme to determine the shock path in a region where the folds yield triple values u for some x (i.e., the case shown in Figure J.4). This scheme depends on the following operations that require nonlinear solvers: 1. Detection of Fold Edges. Let acritical be the value found at the break time of the shock, then 

aedge,1 aedge,2

 = EDGE (acritical )

where   ∂x  ∂x  = 0 = ∂a aedge,1 ∂a aedge,2

and

aedge,1 < acritical < aedge,2

(J.13)

776

Appendix J: Additional Details and Fortification for Chapter 10

2. Root Finding for a. Let xg be in a region where three different values of u correspond to one value of x and s. ⎞ aˆ 1 ⎝ aˆ 2 ⎠ = FINDa (xg , s) aˆ 3 ⎛

(J.14)

where a1 > a2 > a3

and

x(a1 , s) = x(a2 , s) = x(a3 , s) = xg

3. Evaluation of Net Area.  I(y) =

a3 (y)

a1 (y)

' ( ∂x u(s, a) da ∂a

(J.15)

where a1 (y) and a3 (y) are found using the operation FINDa(y).

Shock-Fitting Scheme: r Given: s break , s and acritical r For s = s break + s, sbreak + 2 s, . . . 1. Calculate xg as the average of the edge values, xg =

    1  x s, aedge,1 + x s, aedge,2 2

where aedge,1 and aedge,2 are found using EDGE (acritical ). 2. Using xg as the initial guess, find xˆ such that I ( xˆ ) = 0 3. xshock (s) ← xˆ Using the shock-fitting scheme on the Burger equation (J.5) subject to the initial condition (J.6), we find two shocks paths, one starting at (t, x) = (29.1, 41) and the other one starting at (t, x) = (120, 54.8), as shown in Figure J.5. One can see that the shock paths for this example are approximately straight lines. Furthermore, we also note that the two shock paths do not intersect with each other. Thus even though the curves shown in Figure J.1 for the case of s = 600 may contain portions in which u has more that three values corresponding to a specific value of x, it does not immediately imply that the two shocks path would intersect. In the next section, we show that the shock paths will need to satisfy jump conditions and that the path being linear is not due to the initial condition but rather due to the coefficient b(x, t, u) = u for the inviscid Burger equation.

Appendix J: Additional Details and Fortification for Chapter 10

777

250

200

Figure J.5. Two shock paths for the Burger equation under the conditions given by (J.6) using the shock-fitting scheme based on the equal-area principle.

t

150

100

50

0 0

50

100

150

x J.1.4 Jump Conditions We further limit our discussion to the case where b(x, t, u) = b(u) in (J.1). Under this condition, the differential equation (J.1) results from (or can be recast as) a conservation equation given by  β       ∂ β u (x, t) dx = flux u(α,t ) − flux u(β,t) + c (x, t, u) dt (J.16) ∂t α α  where flux(u) = b(u)du and c(x, t, u) is the volumetric rate of generation for u. Now suppose at t, α < β is chosen so that the shock discontinuity is at x = xs + located between α and β. Let x− s and xs be the locations slightly to left of and right of xs , respectively. Then   −  β  β  xs     ∂ u (x, t) dx + u (x, t) dx = flux u(α,t ) − flux u(β,t) + c (t, x) dt ∂t α x+ α s (J.17) Applying the Leibnitz rule (5.52) to (J.17), we obtain  x−s  β   dx−  +  dx+ ∂u ∂u s s dx + u x− , t + dx − u xs , t s ∂t dt ∂t dt α x+ s       = flux u(α,t ) − flux u(β,t) +

α

β

c (t, x) dt

+ Next, we take the limit as α → x− s and β → xs . This yields

   dx−   dx+     s s u x− − u x+ = flux u(x−s ,t) − flux u(x+s ,t) s ,t s ,t dt dt  x+ where x−s cdx = 0 if we assume that c(x, t, u) is piecewise continuous.1 As the s previous section showed, the shock propagation is continuous and implies that 1

A more complete assumption for c is that it does contain any Dirac delta distribution (i.e., delta impulses).

200

778

Appendix J: Additional Details and Fortification for Chapter 10 − dx+ s /dt = dxs /dt = dxs /dt. Using the jump notation, arrive at H I flux(u) dxs = dt u!

ξ ! = ξ|u(x−s ,t) − ξ|u(x+s ,t) , we (J.18)

which is known as the Rankine-Hugoniot jump conditions.2 This condition equates the shock speed dxs /dt to the ratio of jump values of the flux(u) and u. This can be used to help find the next position of the discontinuity for some simple cases; that is, the shock path can be found using (J.18) without using the equal area approach discussed in the previous section. Furthermore, the jump condition can be used to eliminate some shock solutions that satisfy the partial differential equations on the piecewise continuous region, but nonetheless would violate the Rankine-Hugoniot conditions.

EXAMPLE J.1.

Consider the inviscid Burger’s equation

∂u ∂u +u =0 ∂t ∂x subject to the discontinuous condition + 1 if x ≤ a u(x, 0) = 0 if x > a For this problem, b(u) = u, and the flux is  u2 flux(u) = u du = 2 Because the initial condition is immediately discontinuous, the break time in this case is at t = 0. Using the Rankine-Hugoniot jump condition (J.18), J 2 K u dxs u+ + u− 2 = L M = dt 2 u Because u = constant along the characteristics, u− = 1 and u+ = 0, yielding dxs 1 t = → xs = + a dt 2 2 Thus the solution is given by ⎧ t ⎪ ⎪ ⎨ 1 if x ≤ 2 + a u(x, t) = ⎪ t ⎪ ⎩ 0 if x > + a 2 2

If the conservation equation (J.16) is given in a more general form by  β  β      ∂ φ (x, t, u) dx = flux α, t, u(α,t ) − flux β, t, u(β,t) + c (x, t, u) dt ∂t α α the Rankine-Hugoniot condition (J.18) should be replaced instead by H I flux (x, t, u) dxs I = H dt φ (x, t, u)

Appendix J: Additional Details and Fortification for Chapter 10

The jump conditions given in (J.18) will generally not guarantee a unique solution. Instead, additional conditions known as admissibility conditions, more popularly known as Lax entropy conditions, are needed to achieve physical significance and uniqueness. We now state without proof the following condition known as the Lax entropy conditions applicable to the case where flux(u) is convex, that is, d2 flux/du2 > 0:   d flux  dxs d flux  ≥ ≥ (J.19) du u=u− dt du u=u+ Thus these conditions put the necessary bounds on the shock speed, at least for the case of convex fluxes.3 This condition simply implies that if the characteristics appear to be intersecting in the direction of decreasing t (time reversal), then this solution is not admissible.

EXAMPLE J.2.

For the inviscid Burger equation and initial condition given by + ∂u ∂u A for x ≤ 0 +u =0 u(x, 0) = B for x > 0 ∂t ∂x

where A < B. Let 0 < m < 1, then a solution that contains two shock paths given by ⎧ for x ≤ (A + m)t/2 ⎨ A u(x, t) = m for (A + m)t/2 < x ≤ (m + B)t/2 (J.20) ⎩ B for x > (m + B)/2 will satisfy the Rankine-Hugoniot jump conditions at both regions of discontinuities. This means there are an infinite number of possible solutions that will satisfy the differential equation and jump discontinuity conditions. However, using the entropy conditions given in (J.19), we obtain dxs >B dt which is not true (because it was a given in the initial condition that A < B). This means that the discontinuous solutions in (J.20) are inadmissible based on the entropy conditions. We see in the next section that the rarefaction solution turns out to be the required solution. A>

J.1.5 Rarefaction When a first-order quasilinear PDE is coupled with a discontinuous initial condition, we call this problem a Riemann problem. We already met these types of problems in previous sections. In Example J.1, we saw that the Riemann problem there resulted in a shock propagated solution for the inviscid Burger equation, where u(x ≤ a, 0) = 1 and u(x > a, 0) = 0. However, if the conditions were switched, that is, with u(x ≤ a, 0) = 0 and u(x > a, 0) = 1, the method of characteristics will leave a domain in the (x, t) plane without specific characteristic curves, as shown in Figure J.6.4 In contrast 3 4

A set of more general conditions are given by Oleinik entropy conditions, which are derived using the approach known as the vanishing viscosity methods. If the initial condition were not discontinuous, this would have been filled in without any problem, especially because the characteristics would not even intersect and no shocks would occur.

779

780

Appendix J: Additional Details and Fortification for Chapter 10

u(x,t)=? t Figure J.6. Rarefaction in a Riemann problem. 0

a

x

to the shock-fitting problem, this case is called the rarefaction, a term that originates from the phenomenon involving wave expansion of gases. We limit our discussion to the case of (J.1), where b(x, t, u) = b(u) and c(x, t, u) = 0 with the additional assumption that the inverse function b−1 (·) can be obtained. Consider ∂u ∂u + b(u) =0 ∂t ∂x

(J.21)

subject to 5 u(x, 0) =

uleft right

u

if x ≤ a if x > a

(J.22)

    where b uleft < b uright . Let the initial data be parameterized by ξ, that is, at s = 0, t(s = 0) = 0, x(s = 0) = ξ and u(ξ, 0) = uleft or u(ξ, 0) = uright when ξ ≤ a or ξ > a, respectively. Then the characteristics are given by ⎧  left  t+ξ if ξ ≤ a ⎨ b u x = b (u(ξ, 0)) t + ξ → x = ⎩  right  b u t + ξ if ξ > a Rarefaction will start at x = a when t = 0. The characteristics at this point can be rearranged to be   −1 x − a u(a, 0) = lim b (x,t)→(a,0) t We could pose that the solution in the rarefaction domain to be of the form   x−a u(x, t) = b−1 t and see that this will satisfy the differential equation, that is,     ∂u ∂u 1 x−a x−a d x−a + b(u) =0 → − + b−1 =0 ∂t ∂x t t t d ((x − a)/t) t The solution of (J.21) subject to (J.22) is then given by   ⎧ left u if x ≤ b uleft t + a ⎪ ⎪ ⎪   ⎨     −1 x − a b if b uleft t + a < x ≤ b uright t + a u(x, t) = ⎪ t ⎪ ⎪   ⎩ right u if x > b uright t + a It is left as an exercise (E10.20) to show that (J.23) is piecewise continuous.

(J.23)

Appendix J: Additional Details and Fortification for Chapter 10

781

For the inviscid Burger equation and initial conditions given by, + ∂u ∂u 0.5 if x ≤ 2 +u = 0 subject to u(x, 0) = 1.5 if x > 2 ∂t ∂x

EXAMPLE J.3.

the rarefaction solution becomes ⎧ 0.5 ⎪ ⎪ ⎪ ⎨ x−2 u(x, t) = ⎪ t ⎪ ⎪ ⎩ 1.5

if x ≤ 0.5t + 2 if 0.5t + 2 < x ≤ 1.5t + 2 if x > 1.5t + 2

J.2 Classification of Second-Order Semilinear Equations: n > 2 When the number of independent variables is more than two, the principal part of the semilinear equation is given by the following general form: F prin =

n  n 

αi,j (x) μi,j

(J.24)

i=1 j =1

Just as we did in the previous section, we look for a new set of independent variables {ξ1 , . . . , ξn }, such that under the new coordinates, Fˆ prin (ξ1 , . . . , ξn ) =

n 

(ξ)

i μi,i

where

 = 0, − 1 or + 1

(J.25)

i=1

where we use the following notation: (ξ)

=

∂u ∂ξi

μi,j

(ξ)

=

∂2u , ∂ξi ∂ξ j

(ξ)

=

∂3u , ∂ξi ∂ξ j ∂ξk

μi

μi,j,k

1 ≤ i, j ≤ n 1 ≤ i, j, k ≤ n

.. .

(J.26)

The classification of these forms is then given in the following definition: Definition J.1. The canonical forms of second-order semilinear equations given by n    (ξ) (ξ) (J.27) i μi,i = f ξ1 , . . . , ξn , u, μ1 , . . . , μ(ξ) n i=1

are classified to be elliptic, parabolic, hyperbolic, and ultrahyperbolic according to the following conditions: Elliptic: Parabolic: Hyperbolic: Ultra-Hyperbolic:

if i if i if i if i

= 0 all have the same sign = 0 for some 1 ≤ i ≤ n = 0 all have the same sign except for one = 0 and a ≥ b > 0 , c ≤ d < 0 for a = b = c = d

782

Appendix J: Additional Details and Fortification for Chapter 10

Unfortunately, finding a change in coordinates, ξi = ξi (x1 , x2 , . . . , xn )

i = 1, 2, . . . , n

(J.28)

that would yield the canonical forms (J.27) may not be always be possible. However, when the coefficients in the principal parts are constants, then the equation can be transformed into the canonical forms given in Definition J.1. THEOREM J.1.

Consider the second-order semilinear equation given by n  n 

Ai,j μi,j = f (x, u, μ1 , . . . , μn )

(J.29)

i=1 j =1

where Ai,j = Aj,i are constants. Let (ξ1 , ξ2 , . . . , ξn ) be a set of new independent variables defined by ⎛ ⎞ ⎞ ⎛ x1 ξ1 ⎜ x2 ⎟ ⎜ ξ2 ⎟ ⎜ ⎟ ⎟ ⎜ (J.30) ⎜ .. ⎟ = DU ⎜ .. ⎟ ⎝ . ⎠ ⎝ . ⎠ ξn

xn

where, U is an orthogonal matrix such that UAU T = , where  = diag(λ1 , λ2 , . . . , λn ) is the diagonal matrix of eigenvalues of A and D = diag (d1 , d2 , . . . , dn ), where ⎧ √ ⎨ 1/ |λi | if λ1 = 0 di = ⎩ 0 if λi = 0 and λi is the ith eigenvalue of A. Then under the change of coordinates given by (J.30), the partial differential equation (J.29) becomes n 

  (ξ) (ξ) i μi,i = fˆ ξ1 , . . . , ξn , u, μ1 , . . . , μ(ξ) n

, i = 0, − 1 or 1

(J.31)

i=1

PROOF.

With (J.30), the partial differential operators ∂/∂xi are ⎛ ⎜ ⎜ ⎜ ⎝

∂/∂x1 ∂/∂x2 .. . ∂/∂xn

⎞

⎛

⎟ ⎜ ⎟ ⎜ ⎟ = UT D ⎜ ⎠ ⎝

∂/∂ξ1 ∂/∂ξ2 .. .

⎞ ⎟ ⎟ ⎟ ⎠

∂/∂ξn

Using the partial differential operators, the partial differential equation (J.29) can written as ⎛ ⎞ ∂/∂x1 ⎟   ⎜ ⎜ ∂/∂x2 ⎟ ∂/∂x1 ∂/∂x2 · · · ∂/∂xn A ⎜ ⎟ u = f (x, u, μ1 , . . . , μn ) .. ⎝ ⎠ . ∂/∂xn

Appendix J: Additional Details and Fortification for Chapter 10

783

or ⎞ ∂/∂ξ1  ⎟ ⎜ .. DUAU T D ⎝ ⎠ u = f (x, u, μ1 , . . . , μn ) . ∂/∂ξn ⎛



∂/∂ξ1

···

∂/∂ξn

which can then be simplified to be n 

  (ξ) (ξ) sign (λi ) μi,i = fˆ ξ1 , . . . , ξn , u, μ1 , . . . , μ(ξ) n

i=1

where ⎧ ⎨ +1 0 sign(λi ) = ⎩ −1

if λi > 0 if λi = 0 if λi < 0

Consider the second-order differential equation with three independent variables x, y, and z,

EXAMPLE J.4.

3

∂2u ∂2u ∂2u ∂2u ∂2u ∂2u + 5 − 2 + + 2 + 3 = ku ∂x2 ∂x∂y ∂x∂z ∂y2 ∂y∂z ∂z2

(J.32)

We now look for new coordinates ξ1 , ξ2 , and ξ3 that would transform (J.32) into the canonical form given in (J.27) for purposes of classification. Extracting the coefficients into matrix A, ⎛ ⎞ 3 2.5 −1 A = ⎝ 2.5 1 1 ⎠ −1 1 3 Using Schur triangularization, we can obtain the orthogonal matrix U ⎛ ⎞ 0.5436 −0.7770 0.3176 U = ⎝ −0.0153 0.3692 0.9292 ⎠ 0.8392 0.5099 −0.1888 and the diagonal normalizing matrix D, D = diag (0.9294, 0.5412, 0.4591) The new coordinates are obtained as follows ⎞ ⎛ ⎞ ⎛ ⎞ ⎛ 0.5252x − 0.7221y + 0.5029z x ξ1 ⎝ ξ2 ⎠ = DU ⎝ y ⎠ = ⎝ −0.0083x + 0.1998y + 9.5029z ⎠ ξ3 0.3853x + 0.2341y − 0.0867z z As a check, we can apply the change of coordinates while noting that the secondorder derivatives of ξi , e.g., ∂ 2 ξi / (∂x∂y), are zero. Thus  2 3  3   ∂ξi ∂ξ j ∂2u ∂ u = ; for p, q = x, y, z ∂p∂q ∂p ∂q ∂ξi ∂ξ j i=1 j =1

784

Appendix J: Additional Details and Fortification for Chapter 10

When substituted into (J.32), we obtain 3  3  i=1 j =1

ij

∂2u = ku ∂ξi ∂ξ j

where ij = a11

∂2ξj ∂ 2 ξi ∂ 2 ξi + a + · · · + a 12 33 ∂x2 ∂x∂y ∂z2

For instance, 12

=

(3)(0.5052)(−0.083) + (2.5)(0.5052)(0.1998) +(−1)(0.5052)(0.5029) + · · · + (3)(0.2952)(0.5029)

After performing the computations, we find 11 = −1, 22 = 33 = 1 and for i = j , ij = 0, i.e. −

∂2u ∂2u ∂2u + + = ku ∂ξ1 ∂ξ1 ∂ξ2 ∂ξ2 ∂ξ3 ∂ξ3

Thus we can classify (J.32) to be hyperbolic.

J.3 Classification of High-Order Semilinear Equations For partial differential equations that have orders higher than two, the canonical forms are more difficult to fix. Instead, the classification is to indicate whether a solution by characteristics is possible or not. We limit our discussion to cases involving two independent variables. Recall that for the second-order equation with two independent variables given by A(x, y)uxx + B(x, y)ux,y + C(x, y)uy,y = f (x, y, u, ux , uy )

(J.33)

the characteristics were obtained by solving the characteristic form, Q(ξx , ξy ) = A(x, y)ξx2 + B(x, y)ξx ξy + C(x, y)ξy2

(J.34)

Prior to the determination of whether the equation can be transformed to the hyperbolic, elliptic, or parabolic canonical forms, the roots of the characteristic form became critical. For the hyperbolic equations, the roots were real. For the parabolic equations, the roots were equal. And for the elliptic equations, the roots were complex. By using the character of the roots, we can then extend the concept of hyperbolic, parabolic, and elliptic to higher orders. Definition J.2. For an mth-order semilinear partial differential equation in two independent variables x and y, m  i=0

Ai (x, y)

  ∂mu = f x, y, u, μ[1] , . . . , μ[m−1] ∂ i x∂ m−i y

(J.35)

Appendix J: Additional Details and Fortification for Chapter 10

785

the characteristic form is given by m m       Q ξ x , ξy = ξx − ri (x, y)ξy Ai (x, y)ξxi ξym−i = i=0

(J.36)

i=0

where ri (x, y), i = 1, 2, . . . , m are the roots of the polynomial m 

Ai (x, y) ri = 0

(J.37)

i=0

Then at a fixed point (x, y), equation (J.35) is classified as Hyperbolic: Parabolic: Elliptic: Mixed:

if all the roots ri are real and distinct if all the roots ri are equal if all the roots ri are complex otherwise

Thus for the hyperbolic case, we can determine m characteristics ξ(i) (x, y) by solving the m characteristic equations given by ξ(i),x − ri (x, y)ξ(i),y = 0

i = 1, 2, . . . , m

(J.38)

that is, solving dx dy =− 1 ri (x, y)

ξ(i) (x, y) = constant

(J.39)

Furthermore, note that if m is an odd number, then the partial differential equation can not be elliptic.

APPENDIX K

Additional Details and Fortification for Chapter 11

K.1 d’Alembert Solutions Having the general solution for the one-dimensional wave equation as given in (11.17), we can start fitting them to initial and boundary conditions. We first consider the case with an infinite x domain and only the initial conditions are specified. The solution for this type of problem is given by a form known as the d’Alembert solution. Next, we consider the case of semi-infinite domain, that is, x ≥ 0, where we extend the applicability of d’Alembert solutions for systems with additional boundary conditions. Finally, we consider the case where the spatial domain is a finite segment, for example, 0 ≤ x ≤ L.

K.1.1 Infinite-Domain Wave Equation with Only Initial Conditions The system is described by ∂2u 1 ∂2u − 2 2 =0 2 ∂x c ∂t subect to

−∞≤x≤∞

u (x, 0) = f (x)

and

∂u (x, 0) = g(x) ∂t

Applying the initial conditions to the general solution for u given in (11.17), u(x, 0)

=

f (x) = φ (x) + ψ (x)

∂u (x, 0) ∂t

=

g(x) = c

dψ dφ −c dx dx

(K.1)

because at t = 0, φ(x + ct) = φ(x) and ψ(x − ct) = ψ(x). Taking the derivative of f (x), df dφ dψ = + dx dx dx Solving (K.1) and (K.2) simultaneously for dφ/dx and dψ/dx, dφ 1 df 1 = + g(x) and dx 2 dx 2c 786

dψ 1 df 1 = − g(x) dx 2 dx 2c

(K.2)

Appendix K: Additional Details and Fortification for Chapter 11 ua

ua

2 1 0 10

Time = 0.0

0 −50

0

50

1

Time = 1.0

0 −50

8

0

50 Time = 3.0

1

6 Time

1

0 −50

4 2

50

0

0 –50

0

50 Time = 8.0

1

0 −50

x

787

0

50

x

Figure K.1. A surface plot of the trajectories of ua (left) and a set of four snapshots of ua at different time instants (right) for the d’Alembert’s solution based on zero initial velocity.

and φ(x) ψ(x)

= =

1 1 f (x) + 2 2c 1 1 f (x) − 2 2c



x

g(τ)dτ + κ1

0



x

g(τ)dτ + κ2

0

However, κ1 = −κ2 because f (0) = φ(0) + ψ(0). Returning to (11.17), u(x, t) =

1 1 [ f (x − ct) + f (x + ct)] + 2 2c



x+ct

g(τ)dτ

(K.3)

x−ct

Equation (K.3) is known as the d’Alembert’s solution of the initial value problem.

EXAMPLE K.1.

Let c = 3, g(x) = sech(x), and f (x) =

4 

ζ (αi , βi , γi , x), where

i=1

ζ (α, β, γ, x) =

 γ 1 + tanh (αx + β) 2

and

i

αi

βi

γi

1 2 3 4

1 1 1 1

4 −4 4 10

1 −1 −0.5 0.5

 1 1 x+ct g(s)ds. (f (x + ct) + f (x − ct)) and ub(x, t) = 2 2c x−ct From Figures K.1, we see that the initial distribution given by f (x) is gradually split into two shapes that are both half the height of the original distribution. Both shapes move at constant speed equal to c but travel in the opposite directions. However, for ub, we see from Figures K.2 that the influence of the initial velocities is propagated within a triangular area determined by speed c. Combining both effects, the solution u = ua + ub is shown in Figures K.3.

Let ua (x, t) =

788

Appendix K: Additional Details and Fortification for Chapter 11 ub

ub

2 1

0 10

Time = 0.0

1

0 −50

0

0 −50

8

50 Time = 1.0

1

0

50 Time = 3.0

1 6 Time

0 −50

4

0

Time = 8.0

1

2

50

50

0

0 −50

x

0 −50

0

50

x

Figure K.2. A surface plot of the trajectories of ub (left) and a set of four snapshots of ub at different time instants (right) for the d’Alembert’s solution based on zero initial distribution.

K.1.2 Semi-Infinite Domain Wave Equation with Dirichlet Boundary Conditions The equations are given by

u (x, 0) ∂u (x, 0) ∂t

=

⎫ f (x) ⎬

=

g(x) ⎭

∂2u 1 ∂2u − ∂x2 c2 ∂t2

=

for x ≥ 0

;

0

for

x≥0

u(0, t) = ς(t)

t≥0

(K.4) where, for continuity, ς(0) = f (0) and dς/dt(0) = g(0). We can first find a solution, v(x, t), whose domain is −∞ < x < ∞. The desired solution, u(x, t), will be obtained by restricting v(x, t) values at 0 ≤ x ≤ ∞, that is,  u(x, t) = v(x, t)x≥0

u

u

TIme = 0.0

1

0 −50

2 1 0 10

(K.5)

0

0 −50

8

0

1

Time

6 50

4 2

0 0 –50

0 −50

0 −50

50 Time = 3.0

0

50 Time = 8.0

1

x

50 Time = 1.0

1

0

x

50

Figure K.3. A surface plot of the trajectories (left) and four snapshots of the distribution at different time instants for u = ua + ub.

Appendix K: Additional Details and Fortification for Chapter 11

789

Thus let v be the solution of the extended problem given by ∂2v 1 ∂2v − ∂x2 c2 ∂t2 v (x, 0) ∂v (x, 0) ∂t

=

f e (x)

=

g e (x)

=

−∞≤x≤∞

0

v(0, t) = ς(t)

;

t≥0

where, f e (x) = f (x)

and

g e (x) = g(x)

for x ≥ 0

Note that f e and g e have not been defined completely. The solution for v(x, t) is the d’Alembert’s solution, given by v = φe (x + ct) + ψe (x − ct), where   1 1 s 1 1 s φe (s) = f e (s) + g e (τ)dτ and ψe (s) = f e (s) − g e (τ)dτ 2 2c 0 2 2c 0 For x ≥ 0, we have (x + ct) > 0 and so φe (x + ct) is immediately given by  1 1 x+ct φe (x + ct) = f (x + ct) + g(τ)dτ 2 2c 0 However, because (x − ct) < 0 when x < ct, ψe (x − ct) has to be handled differently because f e (s < 0) and g e (s < 0) has not been defined. At x = 0, we have v(0, t) = φe (ct) + ψe (−ct) = ς(t) or

 s ψe (s) = ς − − φe (−s) c

→

 x ψe (x − ct) = ς t − − φe (ct − x) c

Combining the results, and restricting the domain to x ≥ 0, ⎧   1 1 ct+x ⎪ ⎪ f + x) − f − x) + g(τ)dτ (ct (ct ⎪ ⎪ ⎪ 2 2c ct−x ⎪ ⎪ ⎪  ⎪ ⎪ x ⎪ ⎨ +ς t− for 0 ≤ x < ct c u(x, t) = ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪  ⎪  ⎪ 1 1 x+ct ⎪ ⎪ f (x − ct) + f (x + ct) + g(τ)dτ for x ≥ ct ⎩ 2 2c x−ct

(K.6)

K.1.3 Semi-Infinite Wave Equation with Nonhomogeneous Neumann Conditions The equations are given by

u (x, 0) ∂u (x, 0) ∂t

= =

∂2u 1 ∂2u − 2 2 2 ∂x c ∂t ⎫ f (x) ⎬ for x ≥ 0 g(x) ⎭

= ;

0

x≥0 u(0, t) = ς(t)

t≥0

(K.7)

790

Appendix K: Additional Details and Fortification for Chapter 11

df (0). Again, we solve the following extended problem dx but this time with the Neumann boundary condition,

where, for continuity, ϑ(0) =

∂2v 1 ∂2v − ∂x2 c2 ∂t2 v (x, 0) ∂v (x, 0) ∂t

=

f e (x)

=

g e (x)

=

−∞≤x≤∞

0

∂v (0, t) = ϑ(t) ∂x

;

with f e (x ≥ 0) = f (x) and g e (x ≥ 0) = g(x). As before, we have v = φe (x + ct) + ψe (x − ct), where   1 1 s 1 1 s φe (s) = f e (s) + g e (τ)dτ and ψe (s) = f e (s) − g e (τ)dτ 2 2c 0 2 2c 0 Because (x + ct) > 0, φe (x + ct) =

1 1 f (x + ct) + 2 2c



x+ct

g(τ)dτ 0

However, for ψe (x − ct), we can use the Neumann condition to handle the range 0 ≤ x < ct, ∂v (0, t) = ϑ(t) = φe (ct) + ψe (−ct) ∂x from which



 s ψe (s) = ϑ − − φe (−s) → c

ψe (s)

=

−c

ψe (x − ct)

=

−c

−s/c

ϑ(τ)dτ − φe (−s)

0 t−(x/c) 0

ϑ(τ)dτ − φe (ct − x)

Combining the results, while restricting the solution to x ≥ 0,  ⎧  1 1 x+ct ⎪ ⎪ f (ct + x) − f (ct − x) + g(τ)dτ ⎪ ⎪ 2 2c ct−x ⎪ ⎪  ⎪ t−(x/c) ⎪ ⎪ ⎪ ⎨ −c ϑ(τ)dτ for 0 ≤ x ≤ ct 0 u(x, t) = (K.8) ⎪ ⎪ ⎪ ⎪ ⎪ ⎪  ⎪  ⎪ ⎪ 1 x+ct 1 ⎪ ⎩ f (x − ct) + f (x + ct) + g(τ)dτ for x ≥ ct 2 2c x−ct

K.1.4 Wave Equation in Finite Domain We consider only the special homogeneous Dirichlet condition for 0 ≤ x ≤ L < ∞. The equations are given by

u (x, 0) ∂u (x, 0) ∂t

=

⎫ f (x) ⎬

=

g(x) ⎭

1 ∂2u ∂2u − ∂x2 c2 ∂t2 for 0 ≤ x ≤ L

= ;

0

for u(0, t) = 0

0≤x≤L t≥0

(K.9)

Appendix K: Additional Details and Fortification for Chapter 11

where, for continuity, we need f (0) = 0 = f (L). For this case, we use the method of reflection given by the following extension, ∂2v 1 ∂2v − =0 ∂x2 c2 ∂t2 v (x, 0) ∂v (x, 0) ∂t

=

f e (x)

=

g e (x)

−∞ ≤ x ≤ ∞ v(0, t) = 0

with f e and g e both extended to be odd periodic functions, that is, ⎧ f (x) for 0 ≤ x ≤ L ⎨ −f (−x) for − L ≤ x ≤ 0 f e (x) = ⎩ |x| > L f e (x − 2L) The solution can then given by

  u(x, t) = v(x, t)

(K.10)

x≥0

where v(x, t) =

1 1 ( f e (x + ct) + f e (x − ct)) + 2 2c



x+ct

g e (τ)dτ x−ct

K.2 Proofs of Lemmas and Theorems in Chapter 11 K.2.1 Proof for Solution of Reducible Linear PDE, Theorem 11.1 First, let m = 2. Substituting (11.12), while using the commutativity between L1 and L2 given in (11.11), Lu

=

L1 L2 (α1 u1 + α2 u2 ) = L1 (α1 L2 u1 + α2 L2 u2 )

=

α1 L1 L2 u1 = α1 L2 (L1 u1 ) = 0

Next, assume the theorem is true for m =  − 1. Then with L = LAL = L LA where  −1 LA = −1 i=1 Li whose solution is given by uA = i=1 αi ui , and with u = uA + α u , we have Lu

=

LAL (uA + α u ) = LA (L uA + α L u )

=

LAL uA = L (LAuA) = 0

Then, by induction we have proven that (11.12) is a solution for the case when Li = Lj for i = j , i, j = 1, . . . , m. For the case where Li is repeated k times, note that Lki (g j ui )

=

k  =0

= Thus

k! ui L g j Lk− i (k − )!! i

ui Lki g j = 0

⎡⎛ ⎞ ⎤ k  Lki ⎣⎝ g j ⎠ ui ⎦ = 0 j =1

791

792

Appendix K: Additional Details and Fortification for Chapter 11

K.2.2 Proof of Sturm-Liouville Theorem, Theorem 11.2 We begin with the following identity, where φn = φm :  ' ( ' ( ' ( d dφn dφm d dφn d dφm p (x) φm − φn = φm p (x) − φn p (x) dx dx dx dx dx dx dx Using (11.51) to substitute for terms on the right-hand side, we get dz(x) = (λn − λm ) r(x)φn φm dx where,

' ( dφn dφm z(x) = p (x) φm − φn dx dx

Integrating both sides, z(B) − z(A) = λn − λm



B

r(x)φn φm dx

A

Functions φn and φm both satisfy the boundary condition at x = B, which we could write in matrix form as     0 βB (K.11) = B γB 0 where

 B=

φm (B)

dφm /dx(B)

φn (B)

dφn /dx(B)



Because, in a Sturm-Liouville system, βB and γB are not allowed to both be zero, (K.11) has a solution only if the determinant of matrix B is zero. This implies ' ( dφn dφm z(B) = p (B) φm (B) (B) − φn (B) (B) = p (B)det (B) = 0 dx dx The same argument follows through with the boundary condition at x = A, which implies z(A) = 0. Thus we have for λm = λn ,  B r(x)φn (x)φm (x)dx = 0 for m = n A

K.2.3 Proof of Similarity Transformation Method, Theorem 11.3 Assuming symmetry is admitted based on the similarity transformations & t = λ−α t, −β −γ u = λ u, we have & x = λ x and &   F λβ x, λα t, λγ u, . . . , λγ−κα−(m−κ)β μ[κ,m−κ] . . . , = 0 (K.12) where μ[κ,m−κ] =

u ∂ m& t) (∂ m−κ& x) (∂ κ&

Appendix K: Additional Details and Fortification for Chapter 11

After taking the derivative with respect to λ and then setting λ = 1, we obtain a quasilinear differential equation given by   ∂F ∂F ∂F ∂F β& x + α& t + γ& u + · · · + γ − κα − (m − κ)β μ[κ,m−κ] [κ,m−κ] + · · · = 0 ∂& x ∂& t ∂& u ∂μ where the other terms include the partial derivatives of F with respect to the partial derivatives ∂& u/∂& t, ∂& u/∂x, etc. Method of characteristics yields the following equations: d& x d& t d& u dμ[κ,m−κ] dF  = = = ···  = ··· = & β& x αt γ& u 0 γ − κα − (m − κ)β μ[κ,m−κ] At this point, we assume that α = 1 for brevity.1 Solving the first equations excluding the last term will yield the following invariants d& x d& t = & t β& x

→

ζ=

d& t d& u = & t γ& u

→

ψ=

& x & tβ & u & tγ

.. . d& t dμ[κ,m−κ]  = & t γ − κ − (m − κ)β μ[κ,m−κ]

→

φκ,m =

μ[κ,m−κ] & t(γ−κ−(m−κ)β)

.. . plus F , which is another invariant. We also can now use x, t, and u instead of& x,& t, and & u because the invariants also satisfy the symmetry conditions. The general solution of the quasilinear equation can now be given by F = g (ζ, ψ, . . . , φκ,m , . . .) = 0 For the invariants with κ = 0, that is, the partial derivatives with respect to x only, we have μ[0,m] =

∂ m u  γ−mβ  dm ψ = t ∂xm dζm

→

φ0,m =

dm ψ dζm

With μ[κ,m−κ] =

∂ κ  [0,m−κ]  μ ∂tκ

one can show by induction that μ[κ,m−κ] = tγ−κ−(m−κ)β

κ  j =0

1

c j ζj

dm−κ+j ψ dζm−κ+j

→

φκ,m =

κ 

c j ζj

j =0

If α = 0, then we could set β = 1 and proceed with the role of t replaced by x.

dm−κ+j ψ dζm−κ+j

793

794

Appendix K: Additional Details and Fortification for Chapter 11

where c j are simply constants that depend on j , m, κ, β, and γ whose complicated forms are not needed for the purpose of this proof. Thus we conclude that because the invariants φκ,m are just functions hκ,m of ζ, ψ and derivatives of ψ(ζ), we have shown that       dψ g ζ, ψ, . . . , φκ,m , . . . = g ζ, ψ, . . . , hκ,m ζ, ψ, ,... ,... = 0 dζ is a nonlinear ordinary differential equation for ψ(ζ).

APPENDIX L

Additional Details and Fortification for Chapter 12

L.1 The Fast Fourier Transform In this appendix, we obtain matrix representations of the discrete Fourier transforms, which is often used to find the Fourier series through the use of numerical integration methods. For a periodic function g(t) with period T , we have the complex form of the Fourier series defined by   ∞  2πikt Ck exp (L.1) g FS (t) = T k=−∞ √ where i = −1. The Fourier coefficients C can be evaluated by first setting g FS (t) = g(t) and then multiplying (L.1) by exp(−2πi/T ), followed by an integration with respect to t from 0 to T ,      T  T ∞  2πit 2πi(k − )t g(t) exp − dt = Ck exp dt T T 0 0 k=−∞

Because e2mπi = cos (2mπ) + i sin (2mπ) = 1 we have  T 0

with m an integer

5    T 2πi(k − )t T 2πi(k−) dt = exp e −1 = T 2πi (k − ) 0



Thus 1 C = T

 0

T

  2πi g(t) exp − t dt T

if k =  if k = 

(L.2)

Now suppose the function g(t), t ∈ [0, T ], is represented by (N + 1) uniformly distributed points, that is, g 0 , . . . , g N , with g k = g(k t), t = tk+1 − tk , and T = N t. Using the trapezoidal approximation of the integral in (L.2), we have the discretized version given by     N−1  1 2πk g0 + gN

t + i t g k exp − C = N t 2 N k=1

795

796

Appendix L: Additional Details and Fortification for Chapter 12

Now let y = NC and

⎧ ⎨ g0 + gN 2 xk = ⎩ g k−1

for k = 1

(L.3)

for k = 2, . . . , N

then we obtain y =

N 

(k−1)(−1)

xk W[N]

k = , . . . , N

(L.4)

k=1

where W[N] = e(−2π/N)i . Equation (L.4) is known as the discrete Fourier transform of vector x = (x1 , . . . , xN )T . For the determination of y ,  = 1, . . . , N, a matrix representation of (L.4) is given by y = F [N] x where

⎛

F [N]

1 ⎜ 1 ⎜ ⎜ ⎜ =⎜ 1 ⎜ . ⎜ . ⎝ . 1

(L.5) ⎞

1 W[N]

··· ···

1 N−1 W[N]

2 W[N] .. . N−1 W[N]

··· .. .

W[N] .. . (N−1)(N−1) W[N]

···

2(N−1)

⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠

(L.6)

For the special case of N = 2m for some integer m ≥ 1, we can obtain the classic algorithm known as the Radix-2 Fast Fourier Transform, or often simply called Fast Fourier Transform FFT. The FFT algorithm significantly reduces the number of operations in the evaluation of (L.5). First, note from (L.6) that F [1] = 1. For N = 2m , m ≥ 1, we can separate the odd 2 = W[N/2] to obtain a rearrangement of and even indices and use the fact that W[N] (L.4) as follows: y

=

N/2      (2k−2)(−1) (2k−1)(−1) x2k−1 W[N] + x2k W[N] k=1

=

N/2 

N/2  (k−1)(−1)   (k−1/2)(−1) 2 2 x2k−1 W[N] + x2k W[N]

k=1

=

N/2 

k=1 (k−1)(−1)

x2k−1 W[N/2]

k=1

−1 + W[N]

N/2 

(k−1)(−1)

x2k W[N/2]

(L.7)

k=1

 T Equation (L.7) is known as the Danielson-Lanczos equation. Let y = yTA yTB T  N where yA = (y1 , . . . , yN/2 )T and yB = y(N/2)+1 , . . . , yN . Because W[N] = 1 and N/2

W[N] = −1, for  = 1, . . . , N/2, yA

=

odd even F [N/2] P[N] x + [N/2] F [N/2] P[N] x

yB

=

odd even F [N/2] P[N] x − [N/2] F [N/2] P[N] x

Appendix L: Additional Details and Fortification for Chapter 12

797

where odd P[N] =



e1

...

e3

eN−1 ⎛

⎜ ⎜

[N/2] = ⎜ ⎜ ⎝

T

even P[N] =

;

1 W[N] ..

(N/2)−1

(o|e)

...

eN

T

⎞ ⎟ ⎟ ⎟ ⎟ ⎠

⎞

  ⎟ (o|e) (o|e) ⎟P ⎠ [N] = Z[N] I2 ⊗ F [N/2] P[N]

⎛

P[N]

.

e4

W[N]

Comparing with (L.5), we have ⎛

[N/2] F [N/2] ⎜ F [N/2] F [N] = ⎜ ⎝ F [N/2] − [N/2] F [N/2] ⎞ odd P ⎜ [N] ⎟ ⎟ =⎜ ⎝ even ⎠ P[N]

e2

0

0

where



⎛ and

Z[N]

⎜ IN/2 =⎜ ⎝ IN/2

(L.8)

⎞

[N/2] ⎟ ⎟ ⎠ − [N/2]

Using the identities AC ⊗ BD = (A ⊗ B)(C ⊗ D) and A ⊗ (B ⊗ C) = (A ⊗ B) ⊗ C, we have   (o|e) I2 ⊗ F [N/2] = I2 ⊗ Z[N/2] (I2 ⊗ F [N/4] ) P[N/2]       (o|e) I2 ⊗ P[N/2] = I2 ⊗ Z[N/2] I2 ⊗ F [N/4]     (o|e) = I2 ⊗ Z[N/2] I4 ⊗ F [N/4] I2 ⊗ P[N/2] Continuing the recursion we obtain, with F [1] = 1, N = 2m , bitreverse F [N] = G[N] P[N]

(L.9)

where, G[N]

=

bitreverse P[N]

=

     Z[N] I2 ⊗ Z[N/2] I4 ⊗ Z[N/4] · · · IN/2 ⊗ Z[2]      (o|e) (o|e) (o|e) (o|e) IN/2 ⊗ P[2] · · · I4 ⊗ P[N/4] I2 ⊗ P[N/2] P[N]

bitreverse It can be shown that the effect of P[N] on x is to rearrange the elements of x by reversing the bits of the binary number equivalent of the indices. To illustrate, let N = 8, then ⎛ ⎞ ⎞ ⎛ ⎞⎛ 1 0 0 0 0 0 0 0 x1 1 0 0 0 0 0 0 0 ⎜x5 ⎟ ⎜0 0 1 0 0 0 0 0⎟ ⎜0 0 1 0 0 0 0 0⎟ ⎜ ⎟ ⎟ ⎜ ⎟⎜ ⎜x3 ⎟ ⎜0 1 0 0 0 0 0 0⎟ ⎜0 0 0 0 1 0 0 0⎟ ⎜ ⎟ ⎟ ⎜ ⎟⎜ ⎜x7 ⎟ ⎟ ⎜ ⎜ ⎟ 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 bitreverse ⎟ ⎜ ⎜ ⎟ ⎟ x=⎜ x=⎜ P[8] ⎜x2 ⎟ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜0 0 0 0 1 0 0 0⎟ ⎜0 1 0 0 0 0 0 0⎟ ⎜x6 ⎟ ⎜0 0 0 0 0 0 1 0⎟ ⎜0 0 0 1 0 0 0 0⎟ ⎜ ⎟ ⎟ ⎜ ⎟⎜ ⎝x4 ⎠ ⎝0 0 0 0 0 1 0 0⎠ ⎝0 0 0 0 0 1 0 0⎠ 0 0 0 0 0 0 0 1 x8 0 0 0 0 0 0 0 1

798

Appendix L: Additional Details and Fortification for Chapter 12

Instead of building the permutations, we could look at the bit reversal of the binary equivalents of the indices of x (beginning with index 0), ⎛ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎝

000 001 010 011 100 101 110 111

⎛

⎞

⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎟ −−−−−−−→ ⎜ ⎟ reverse bits ⎜ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎝ ⎠

000 100 010 110 001 101 011 111

⎛

⎞

⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎟ −−−−−→ ⎜ ⎟ decimal ⎜ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎝ ⎠

0 4 2 6 1 5 3 7

⎞

⎛

⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎟ −−−→ ⎜ ⎟ add 1 ⎜ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎝ ⎠

1 5 3 7 2 6 4 8

⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠

In summary, we have the following algorithm:

FFT Algorithm: Given: x[=]2m × 1 y ← Reverse Bits(x) For r= 1, . . . , m  y ← I2m−r ⊗ Z[2r ] y End Remark: The MATLAB command for the FFT function is y=fft(x).

EXAMPLE L.1.

Let ⎧ 5 ⎪ ⎪ ⎪ ⎪ ⎪   ⎪ ⎪ 2π ⎨ t cos t g(t) = 4 20 ⎪ ⎪ ⎪ ⎪ t ⎪ ⎪ ⎪ ⎩ 28 − 10

if t < 20 if 20 ≤ t < 80 if t ≥ 20

Now apply the Fourier series to approximate g(t) for 0 ≤ t ≤ 200 with T = 200 and sampling N = 210 + 1 uniformly distributed data points for g(t). Using x as defined in (L.3) and y = FFT (x), we can obtain a finite series approximation given by g FFT,L =

L  k=−L

Ck W

−kt

  L    1 −kt = Real yk+1 W y1 + 2 N k=1

Note that only the first N/2 = 2m−1 terms of y are useful for the purpose of approximation, i.e. L ≤ N/2. Figure L.1 shows the quality of approximation for L = 10 and L = 25.

Appendix L: Additional Details and Fortification for Chapter 12

L=10

L=25

20

20

10

g(t)

10

g(t)

0

0

−10

−10

−20

−20

0

799

50

100

150

200

0

t

50

100

150

t

Figure L.1. Fourier series approximation of g(t) using L = 10 and L = 25.

L.2 Integration of Complex Functions In this appendix, we briefly describe the notations, definitions, and results from complex function theory. Specifically, we focus on the methods for contour integrations of complex functions.

L.2.1 Analytic Functions and Singular Points Definition L.1. Let z = zre + izim be a complex variable, with zre , zim ∈ R. Then a complex function f (z) = f re (zre , zim ) + i f im (zre , zim ) is analytic (or holomorphic) in a domain D, that is, a connected open set, if for every circle centered at z = z∗ inside D, f (z) can be represented by a Taylor series expanded around z = z∗ , f (z) =

∞ 

αk (z − z∗ )k

(L.10)

k=0

where,

 1 dk f  αk = k! dzk z=z∗

(L.11)

Implicit in the preceding definition is the existence of derivatives, dk f/dzk , for k ≥ 1. One necessary and sufficient condition for analyticity of f (z) is given by the following theorem: A complex function f (z) = f re (zre , zim ) + i f im (zre , zim ) is analytic in D if and only if both real functions f re (zre , zim ) and f im (zre , zim ) are continuously differentiable and

THEOREM L.1.

for all z = zre + izim in D.

∂ f re ∂zre

=

∂ f im ∂zim

∂ f re ∂zim

=

−

∂ f im ∂zre

(L.12)

(L.13)

200

800

Appendix L: Additional Details and Fortification for Chapter 12

The pair of equations (L.12) and (L.13) are known as the Cauchy-Riemann conditions. Some Important Properties of Analytic Functions: Let f (z), f 1 (z) and f 2 (z) be analytic in the same domain D, then 1. Linear combinations of analytic functions are analytic; that is, f sum (z) = α1 f 1 (z) + α2 f 2 (z) is analytic. 2. Products of analytic functions are analytic; that is, f prod (z) = f 1 (z) f 2 (z) is analytic. 3. Division of analytic functions are analytic except at the zeros of the denominator; that is, f div (z) = f 1 (z)/ f 2 (z) is analytic except at the zeros of f 2 (z). 4. Composition of analytic functions are analytic; that is, f comp (z) = f 1 ( f 2 (z)) is analytic. 5. The inverse function, f −1 ( f (z)) = z, is analytic if df/dz = 0 in D and f (z1 ) = f (z2 ) when z1 = z2 . 6. The chain rule is given by d df 2 df 1 [f 2 (f 1 (z))] = dz df 1 dz

(L.14)

Definition L.2. A point zo in domain D is called a singularity or singular point of a complex function f (z) if it is not analytic at z = zo . If f (z) is analytic at z = zo , then zo is called a regular point. The singular points can further be classified as follows: 1. A point zo is a removable singular point if f (z) can be made analytic by defining it at zo . (If limz→zo f (z) is bounded, it can be included in the definition of f (z). Then f (z) can be expanded as a Taylor series around zo . For example, with f (z) = (z − 1)(3z/(z − 1)), the point z = 1 is a removable singularity.) 2. A point zo is an isolated singular point if for some ρ > 0, f (z) is analytic for 0 < |z − zo | < ρ but not analytic at z = z0 . 3. A point zo is a pole  of order k, where k is a positive integer, zo )k f (z) has a removable singularity at z = zo , but if  g 1 (z) = (z − k−1 g 2 (z) = (z − zo ) f (z) does not have a removable singularity at z = zo . If k = 1, then we call it a simple pole. 4. A point zo is an essential singular point if it is an isolated singularity that is not a pole or removable.

L.2.2 Contour Integration of Complex Functions In calculating the closed contour integration of f (z), denoted by 3 IC ( f ) = f (z)dz

(L.15)

C

we are assuming that C is a simple-closed curve; that is, C is a curve that begins and ends at the same point without intersecting itself midway. Furthermore, the line

Appendix L: Additional Details and Fortification for Chapter 12

integral will be calculated by traversing the curve C in the counterclockwise manner (or equivalently, the interior of the simple-closed curve is to the left of C during the path of integration). The interior of curve C defines a domain D(C), which is of the type simply connected, as defined next. Definition L.3. A 2D domain D is called a simply-connected domain if the interior points of every simple-closed curve C in D are also in D. Otherwise, the domain is called a multiply connected domain. In short, a simply connected domain is one that does not contain any holes.1 Because of the presence of several theorems that follow later, we start with a brief outline of the development of techniques for contour integration in the complex plane. 1. We start with Cauchy’s theorem to handle the special case when f (z) is analytic on and inside the closed curve C. 2. In Theorem L.3, we show that even though C is the original contour, a smaller curve C inside C can yield the same contour integral values, as long as f (z) remains analytic on C, C and the annular region between C and C . 3. Having established that the contour used for integration is not unique, we shift the focus instead on specific points and construct small circular contours around these points. This leads to the definition of residues. Theorem L.4 then gives a formula to calculate the residues of poles. 4. Using residues, we can then generalize Cauchy’s theorem, Theorem L.2, to handle cases when curve C encloses n isolated singularities. The result is the residue theorem.

THEOREM L.2.

Cauchy’s Theorem. Let f (z) be analytic on and inside a simple closed

curve C, then 3 f (z)dz = 0

(L.16)

C

PROOF.

With z traversing along the curve C,   dzre dzim dz = +i ds ds ds

or in terms of the components of the unit outward normal vector, n = n re + i n im , dz = (−nim + i n re ) ds because dzre = −n im ds

1

;

dzim = n re ds

For higher dimensional regions, if any simple closed path in D can be shrunk to a point, then D is simply connected.

801

802

Appendix L: Additional Details and Fortification for Chapter 12 H

C

C’

C’

b a

Figure L.2. The curves C, C , and H used for proof of Theorem L.3.

Thus

3

3 f (z)dz

=

( f re + if im ) (−nim + i n re ) ds

C

3

3 =

( f im n re + f re n im ) ds + i

− C

( f re n re − f im n im ) ds C

Using the divergence theorem,      3    ∂ f im ∂ f re ∂ f re ∂ f im f (z)dz = + dzre dzim + i − dzre dzim ∂zre ∂zim ∂zre ∂zim C Because analytic functions satisfy the Cauchy-Riemann conditions, the integrands are both zero.

Let C and C be two simple closed curves where C is strictly inside C. Let f (z) be analytic on curves C and C and in the annular region between C and C . Then 3 3 f (z)dz = f (z)dz (L.17)

THEOREM L.3.

C

C

PROOF.

Based on Figure L.2, we see that the integral based on curve H is given by 3  b  a 3 3 f (z)dz = f (z)dz + f (z)dz + f (z)dz − f (z)dz H

C

a

b

C

However, the path integral from a to b satisfies  a  b f (z)dz = − f (z)dz a

b

Furthermore, because f (z) is analytic in the interior of H (i.e., the annular region 4 between C and C ), Theorem L.2 implies that H f (z)dz = 0. Thus 3 3 f (z)dz = f (z)dz C

C

Theorem L.3 does not constrain how the shrinking of curve C to C occurs except for the conditions given in the theorem. For instance, if f (z) is analytic

Appendix L: Additional Details and Fortification for Chapter 12

throughout the interior of C, then the smaller curve C can be located anywhere inside C. We now shift our focus on point zo and the contours surrounding it.

Definition L.4. For a given point zo and function f (z), let C be a simple closed curve that encircles zo such that zo is the only possible singularity of f (z) inside C; then the residue of f (z) at zo is defined as Reszo (f ) =

1 2πi

3 f (z)dz

(L.18)

C

Note that if f (z) is analytic at the point zo , Reszo = 0. If zo is a singular point of f (z), the residue at zo will be nonzero.2 Using Theorem L.3, we can evaluate residues at the poles of f (z) by choosing C to be a small circle centered around zo .

THEOREM L.4.

Cauchy Integral Representation.3 Let zo be a pole of order k ≥ 1 of

f (z), then Reszo ( f ) =

 1 dk−1  lim k−1 [z − zo ]k f (z) (k − 1)! z→zo dz

(L.19)

First, consider the function h(z) = (z − zo ) , where  is an integer. Let Oρ : |z − zo | = ρ, where ρ > 0. The points on the circle Oρ is given by

PROOF.

z = zo + ρeiθ

for 0 ≤ θ ≤ 2π

and (z − zo ) = ρ ei θ

;

dz = iρeiθ dθ

Thus 3

(z − zo ) dz = iρ+1 Oρ



2π 0

ei(+1)θ dθ =

⎧ ⎨ 2πi ⎩

if  = −1 (L.20)

0

if  = −1

where ρ > 0 is bounded. 2 3

A result known as Morera’s theorem guarantees that if Reszo ( f ) = 0, then f (z) is analytic at a small neighborhood around zo . Note that Theorems L.2 and L.4 are both associated with Cauchy’s name, but the two theorems are not the same. Strictly speaking, Cauchy’s integral representation actually refers only to the case of a simple pole, that is, k = 1.

803

804

Appendix L: Additional Details and Fortification for Chapter 12

Because zo is a pole of order k of f (z), there exists a curve C such that the function g(z) = (z − zo )k f (z) is analytic inside and on a curve C, which includes zo as an interior point; that is, it could be expanded into a Taylor series around zo , (z − zo ) f (z) k

=

g(z) =

∞ 

αn (z − zo )n

n=0

f (z)

∞ 

=

αn (z − zo )n−k

(L.21)

n=0

where αn = lim

z→zo

 1 dn g 1 dn  k = lim f (z) − z ] [z o z→zo n! dzn n! dzn

(L.22)

Based on the definition of the residue, choose the curve C to be a small circle Oρ centered at zo such that f (z) is analytic on and inside the circle Oρ except at zo . This means that the radius of the circle, ρ, must be chosen to be small enough such that, inside Oρ , zo is the only singular point of f (z). Taking the contour integral of (L.21), with substitutions of (L.20) and (L.22), 3 f (z)dz

=

Oρ

∞  n=0

3 αn

(z − zo )n−k dz Oρ

=

2πi αk−1

=

 2πi dk−1  lim k−1 [z − zo ]k f (z) (k − 1)! z→zo dz

Thus Reszo ( f ) =

 1 dk−1  lim k−1 [z − zo ]k f (z) (k − 1)! z→zo dz

We now state a generalization of Theorem L.3. This theorem is very useful for the evaluation of contour integrals in the complex plane. Residue Theorem. Let f (z) be analytic on and inside the closed curve C except for isolated singularities: z ,  = 1, 2, . . . , n. Then the contour integral of f (z) along C is given by 3 n  f (z)dz = 2πi Resz ( f ) (L.23)

THEOREM L.5.

C

=1

We prove the theorem only for n = 2, but the same arguments can be generalized easily for n > 2. Let C1 and C2 be nonintersecting closed curves inside C such that the pole z1 is inside C1 only and the pole z2 is inside C2 only. As shown in Figure L.3, f (z) will be analytic in the curve H as well as in the interior points of H.

PROOF.

Appendix L: Additional Details and Fortification for Chapter 12 H

C

z2 C 2

z1

z2

z1

C1

1

2

Figure L.3. Curves C, C1 , C2 , and H used in Theorem L.5.

Thus 3

3

3

f (z)dz = 0 = H

3

f (z)dz − C

f (z)dz − C1

f (z)dz C2

or 3

3

3

f (z)dz = C

f (z)dz + C1

f (z)dz C2

Using the results of Theorem L.4, 3 f (z)dz = 2πi [ Resz1 (z) + Resz2 (z) ] C

Generalizing the approach to n ≥ 1, 3 f (z)dz = 2πi C

n 

Resz (z)

=1

Note that Theorem L.5 is true whether the isolated singularities are poles or essential singularities. However, we limit our applications only to singularities involving poles. As such, the formula for calculating residues when singularities are poles (cf. Theorem L.4) is used when invoking the method of residues.

L.2.3 Path Integrals with Infinite Limits The method of residues can be applied to calculating path integrals in the complex plane,  f (z)dz

(L.24)

P

where path P is a curve parameterized by a ≤ t ≤ b, that is, P : z = zre (t) + izim (t)

(L.25)

805

806

Appendix L: Additional Details and Fortification for Chapter 12

(aR ,bR )left

P bR

R

right

Figure L.4. The circular arcs left aR ,bR and aR ,bR .

aR right

(aR ,bR )

including the case where |z(t = a)| = ∞

|z(t = b)| = ∞

and

(L.26)

We refer to these paths as infinite paths. Some Technical Issues: 1. Parameterization. Path P will be parameterized using a ≤ t ≤ b. We assume that P does not intersect itself and that the path is bounded, with possible exceptions at the end points. 2. Connecting Arcs. Let aR and bR , with a ≤ aR < bR ≤ b, be values of t such that |P(aR )| = |P(bR )| = R. Then one can connect both P(aR ) and P(bR ) by a circular arc of radius R. (We assume the arc does not to intersect P except at t = aR and t = bR ). We denote the arc by left aR ,bR if the arc starting from aR is to the left of right

path P. Likewise, we denote the arc by aR ,bR if the arc starting from aR is to the right of path P (see Figure L.4). The main idea is to combine either the left or right arc with the subpath, P(aR , bR ), to obtain a simple closed curve from which we can apply the method of residues. 3. Convergence Assumptions. In handling the path integration along the left circular arcs, we assume the following condition: lim R

max

z∈(aR ,bR )left

R→∞

| f (z)| = 0

(L.27)

We refer to (L.27) as the convergence condition in the left arc. Together with the following inequality (also known as Darboux’s inequality),        ≤  f (z) |dz| < 2πR  max | f (z)| (L.28) f (z)dz   (aR ,bR )left

we obtain

z∈(aR ,bR )left

(aR ,bR )left

  lim  R→∞

(aR ,bR )left

  f (z)dz = 0

(L.29)

Similarly, we assume the convergence condition in the right arc given by lim R

R→∞

max

z∈(aR ,bR )right

| f (z)| = 0

(L.30)

Appendix L: Additional Details and Fortification for Chapter 12

and obtain

  lim  R→∞

(aR ,bR )right

  f (z)dz = 0

(L.31)

4. Cauchy Principal Value. With finite limits, the following identity is true: 

 P(aR ,br )

f (z)dz =

 f (z)dz +

P(aR ,0)

f (z)dz P(0,br )

  However, the integral P(aR ,0) f (z)dz or the integral P(0,bR ) f (z)dz, or both, may diverge as aR , bR → ∞, even though the integral  f (z)dz (L.32) PV ( f ) = lim aR ,bR →∞ P(aR ,bR )

 converges. In our calculations of P fdz that follow, we mean the limit calculation of (L.32). The integral in (L.32) is known as the Cauchy principal value of f (z). We now state a theorem that shows how the method of residues can be applied to complex integrations along infinite paths. THEOREM L.6.

Let P(t) be an infinite path that does not pass through any singular

points of f (z). 1. Let z1 , z2 , . . . , zn be the singularities in the region to the left of path P(t), and f (z) satisfies the absolute convergence in the left arc condition given in (L.27), then  f (z)dz = P

n 

Resz ( f )

(L.33)

=1

2. Let zˆ 1 , zˆ 2 , . . . , zˆ m be the singularities in the region to the right of path P(t), and f (z) satisfies the absolute convergence in the right arc condition given in (L.30), then  m  f (z)dz = − Reszˆ  ( f ) (L.34) P

=1

PROOF. Based on Figure L.5, where R is chosen large enough such that the contour formed by the subpath P(aR , bR ) and -(aR , bR )left will contain all the singular points of f (z) that are to the left of P. Then using the theorem of residues,



 P(aR ,bR )

f (z)dz −

(aR ,bR )left

f (z)dz =

=1

As R → ∞, (L.29) then implies  f (z)dz = P

n  =1

n 

Resz ( f )

Resz ( f )

807

808

Appendix L: Additional Details and Fortification for Chapter 12

(aR,bR)left R

zn

z2

bR

z1

Figure L.5. The contour used to prove (L.33) in Theorem L.6.

aR

P(aR,bR)

Likewise, based on Figure L.6, where R is chosen large enough such that the contour formed by the subpath −P(aR , bR ) and (aR , bR )right will contain all the singular points of f (z) that are to the right of P. Then using the theorem of residues,  −

 P(aR ,bR )

f (z)dz +

f (z)dz =

(aR ,bR )right

m 

Reszˆ  ( f )

=1

As R → ∞, (L.31) then implies  f (z)dz = − P

n 

Resz ( f )

=1

Note that the convergence conditions, (L.27) and (L.30), are sufficient conditions that may sometimes be too conservative. In some cases, they could be relaxed. In particular, we have the result known as Jordan’s lemma, which is useful when calculating Fourier transforms and Fourier-Sine/Fourier-Cosine transforms. Let f (z) = g(z)eiωz, where ω > 0, with (aR , bR )left and (aR , bR )right as the semicircle in the upper half and lower half, respectively, of the complex plane, THEOREM L.7.

1. If  lim

R→∞

then

max

z2

aR

(L.35)

  f (z)dz = 0

(L.36)

z∈(aR ,bR )left

  lim  R→∞

(aR ,bR )left

-P(aR,bR)

 |g(z)| = 0

bR

Figure L.6. The contour used to prove (L.34) in Theorem L.6.

zm

z1 (aR,bR)

right

Appendix L: Additional Details and Fortification for Chapter 12

2. If



  f (z)dz = 0

(L.38)

z∈(aR ,bR )right

  lim  R→∞

(aR ,bR )right

PROOF.

(L.37)

max

lim

R→∞

then

 |g(z)| = 0

We show the theorem only for the left arc, that is, upper half of the complex

plane, On the semicircle, we have z = Reiθ . Thus dz = Reiθ dθ

−→

|dz| = R|dθ|

and eiωz

=

eiωR(cos θ+i sin θ)

=

e−ωR sin θ eiωR cos θ

 iωz e  = e−ωR sin θ

−→

Also, note that with 0 ≤ θ ≤ π2 , sin θ ≥ Using these identities and inequality,       ≤  f (z)dz   (aR ,bR )left

(aR ,bR )left

2θ π    g(z) eiwz |dz|

 ≤

max

z∈(aR ,bR )left


0, we can use the closed-contour in the lower region of the complex plane. Doing so, we have  ∞   x3 −iωx √ ( f ) + Res √ (f ) e dx = −2πi Res (L.41) (1−i)/ 2 (−1−i)/ 2 [ ] [ ] 4 −∞ 1 + x

Appendix L: Additional Details and Fortification for Chapter 12

and Res[(1−i)/√2] ( f )

=

'  ( √ 1−i 1 z − f (z) = e(−1−i)ω/ 2 √ √ 4 z→(1−i)/ 2 2

Res[(−1−i)/√2] ( f )

=

'  ( √ −1 − i 1 z − f (z) = e(−1+i)ω/ 2 √ √ 4 z→(−1−i)/ 2 2



∞ −∞

lim

lim '

x3 iωx e dx 1 + x4

=



 ( √ ω −ω/ 2 −πi cos √ e 2

Combining both cases, ' 3 (  ∞   √ x x3 iωx ω F = e dx = −i π cos e[−|ω|/ 2] √ [sgn(ω)] 4 4 1+x 2 −∞ 1 + x

Special Applications and Extensions: 1. Functions Involving Sines and Cosines. Let P(t) = t, with −∞ ≤ t ≤ ∞. When the integrand contains cos(x) or sin(x) in the numerator, the method of residues cannot be used directly because the arc conditions given in (L.27) or (L.30) are no longer satisfied. (For instance, limzim →±∞ | cos(z)| = limzim →±∞ | sin(z)| = ∞). An alternative approach is to use Jordan’s lemma. Because   (L.42) g(x) cos(αx) = Re g(x)eix we could apply the method of residues on the integral in the right hand side of the following equation: ' ∞ (  ∞ g(x) cos(αx)dx = Re g(x)eiαx dx (L.43) −∞

−∞

Similarly, we have 

∞

−∞

' g(x) sin(αx)dx = Im

∞

−∞

( g(x)e dx iαx

(L.44)

Based on Jordan’s lemma, that is, Theorem L.7, with ω = α > 0, we need to satisfy only the condition given in (L.35) and apply it to the contour in the upper region of the complex plane,   max |g(z)| = 0 (L.45) lim R→∞

EXAMPLE L.3.

|z|=R,zim ≥0

Consider the following integral  ∞ 2 x cos x dx 4 −∞ 1 + x

811

812

Appendix L: Additional Details and Fortification for Chapter 12

Using a semicircle in the upper region of the complex plane as the contour of integration, we apply Theorem L.7 to obtain  R   f (zre )dzre = 2πi Res[1+i] ( f ) + Res[−1+i] ( f ) lim R→∞ −R

where, z2 eiz 1 + z4

f (z) = with

Then,



∞

−∞

Res[1+i]/√2 ( f )

=

Res[−1+i]√2 ( f )

=

√ 2(1 − i) (−1+i)/√2 e 8 √ − 2(1 + i) (−1−i)/√2 e 8

'

x2 cos x dx 1 + x4

( x2 eix Re dx 4 −∞ 1 + x '    ( √ π 1 1 − sin √ e(−1/ 2) √ cos √ 2 2 2

= =

∞

2. Rectangular Contours. Sometimes the limits involve a line that is shifted parallel to the real axis or the imaginary axis. In these cases, it may often be convenient to use evaluations already determined for the real line or imaginary axis. To do so, we need a rectangular contour. This is best illustrated by an example.

Let us evaluate the Fourier transform of a Gaussian function,  ∞   ∞  2 −αx2 −αx2 −iωx = F e e e dx = e−αx −iωx dx

EXAMPLE L.4.

−∞

−∞

where α > 0. First, consider ω > 0. We could simplify the integral by first completing the squares,  8   2 9 iω 2 iω iω 2 2 −αx − iωx = −α x + x + − α 2α 2α = thus



∞

−∞

−αx2 −iωx

e

  iω 2 ω2 −α x + − 2α 4α

dx

= =

−ω2 /(4α)



e

e−ω

2

/(4α)



∞ −∞

2

e−α[x+iω/(2α)] dx

∞+iω/(2α) −∞+iω/(2α)

e−αz dz 2

Now consider the rectangular contour shown in Figure L.8.

Appendix L: Additional Details and Fortification for Chapter 12

813

zIm

Figure L.8. A rectangular contour used in Example L.4.

2

zRe

R

Because the function e−αz is analytic throughout the region,  R+iω/(2α)  −R+iω/(2α)  R 2 2 2 e−αz dz + e−αz dz + e−αz dz 2

−R

R

R+iω/(2α)



+

−R

e−αz dz = 0 2

−R+iω/(2α)

Two of the integrals reduces to zero,  R+iω/(2α) 2 lim e−αz dz = 0 R→∞ R

and



−R

lim

e−αz dz = 0 2

R→∞ −R+iω/(2α)

resulting with 

∞+iω/(2α) −∞+iω/(2α)

e−αz dz = 2



∞

−∞

7 e−αz dz = 2

π α

Using a rectangular contour in the lower region, a similar approach can be used to handle ω < 0. Combining all the results, we obtain  2  7π 2 F e−α x = e−ω /(4α) α This says that the Fourier transform of a Gaussian function is another Gaussian function.

3. Path P Contains a Finite Number of Simple Poles. When the path of integration contains simple poles, the path is often modified to avoid the poles using a semicircular indentation having a small radius,  as shown in Figure L.9. Assuming convergence, the calculation for the integral proceeds by taking the limit as  → 0.

P

Figure L.9. (a) Pole zo lies on path P. (b) Path P avoids zo .

z0 (a)

P’ z0 (b)

814

Appendix L: Additional Details and Fortification for Chapter 12 zIm R

Figure L.10. The contour used to solve

∞ −∞

[sin(x)/x]dx.

zRe (-)

(+)

EXAMPLE L.5.

Let us determine the integral  ∞ sin(x) dx x 0

(L.46)

First, we evaluate the integrals with limits from −∞ to ∞. Using the techniques for solving integrals with sinusoids as given in (L.44), ' ∞ ix (  ∞ sin(x) e dx = Im dx x −∞ −∞ x Using the path in the real line, z = 0 is a pole in the real line. Thus, modifying the path to avoid the origin, we obtain the closed contour shown in Figure L.10 given as C = (−) +  + (+) + R . The integral along  can be evaluated by setting z = eiθ . As a consequence, dz = idθ z and

 

eiz dz = z

and taking the limit as  → 0,



lim

→0 

Conversely, we have



0 π

eiz dz = −iπ z

 lim

R→∞ R

Thus



∞

−∞

or



∞ −∞

  exp ieiθ idθ

eiz dz = 0 z

eix dx = iπ x

sin(x) dx = Im [iπ] = π x

Because the function sin(x)/x is an even function, we could just divide the value by 2 to obtain the integral with limits from 0 to ∞, that is,  ∞ sin(x) π dx = (L.47) x 2 0

Appendix L: Additional Details and Fortification for Chapter 12

815

zIm

∞

zn

R

[(x2 + 4) −∞

Figure L.11. The contour used to solve cosh(x)]−1 dx.

2i

z3 z2 z1

zRe P

4. Regions Containing Infinite Number of Poles. In case there is an infinite number of poles in the region inside the contour, we simply extend the summation of the residues to contain all the poles in that region. If the infinite sum of residues converge, then the method of residues will still be valid, that is, 3 ∞  f (z)dz = Resz ( f ) (L.48) C

EXAMPLE L.6.

=1

Let us evaluate the following integral:  ∞  ∞ 1 f (x)dx = dx 2 −∞ −∞ (x + 4) cosh(x)

(L.49)

From the roots of (z2 + 4) and the roots of cosh(z) = cos(iz), the singularities are all simple poles given by: z0 = 2i,

z =

2 − 1 πi, 2

 = 1, 2, . . . , ∞

and their complex conjugates. Using the usual semicircular contour to cover the upper region of the complex plane as shown in Figure L.11, the method of residues yields,   ' (  ∞  f (z)dz − f (z)dz = 2πi Res(2i) [ f ] + Res(z ) [ f ] (L.50) lim R→∞

P

R

=1

Along the path of R , we have z = Reiθ . We find that        1   < lim R−2 exp −Reiθ  = 0 lim  2 i2θ R→∞ (R e + 4) cosh (Reiθ )  R→∞  Thus we have limR→∞ R f (z)dz = 0. As for the residues, 1 1 = z→2i (z + 2i) cosh(z) 4i cos(2)

Res(2i) [ f ] = lim

and with z = i(2 − 1)π/2, together with the application of L’Hospital’s rule, Res(z ) [ f ]

= = =

lim

z→z

z2

(z2

z − z + 4) cosh(z)

−1 1 i sin(iz +4 )

4  (−1)  i 42 − (2 − 1)2 π2

816

Appendix L: Additional Details and Fortification for Chapter 12 z

rb a

b

-1

Figure L.12. Geometric interpretation of ra , rb, θa , and θb given in (L.53).

ra 1

zRe

Combining all these results, we have  ∞ ∞  1 π 1 dx = + 8π (−1) 2 2 2 cos(2) 4 − (2 − 1)2 π2 −∞ (x + 4) cosh(x) =1

(L.51)

5. Integrals along Branch Cuts. When the integrand involves multivalued complex functions, a branch cut is necessary to evaluate the integral. This means that a Riemann sheet4 has to be specified by selecting the range of the arguments of complex variable z. Usually, the ranges for the argument are either 0 < arg(z) < 2π, −π < arg(z) < π, (π/2) < arg(z) < (5π/2) or −π/2 < arg(z) < (3π/2) for branch cuts along the positive real line, negative real line, positive imaginary line, or negative real line, respectively. In other cases, the range of arg(z) may be a finite segment in the complex plane. Once the particular Riemann sheet has been selected, the method of residues can proceed as before. EXAMPLE L.7.

Consider the integral  1 −1 (x2

dx √ + 1) 1 − x2

(L.52)

This is a finite integral in which the integrand contains a square root in the denominator. One can check that the points z = 1 and z = −1 are branch points5 of f (z) where f (z) =

1 √ (z2 + 1) z2 − 1

(Note that we used z2 − 1. The form 1 − x2 will show up from the calculations later.) We could be rewrite the square root terms as " ! z2 − 1 = (z − 1)(z + 1) " = (ra eiθa ) (rbeiθb ) √ = ra rb ei(θa +θb)/2 where, z − 1 = ra eiθa

and

z + 1 = rbeiθb

(L.53)

(see Figure L.12.) 4 5

By Riemann sheet, we simply mean a subdomain that is single-valued. A point zo is branch point of a function f (z) if there exists a closed curve that encircles zo that would yield different evaluations of f (z) after one encirclement.

Appendix L: Additional Details and Fortification for Chapter 12

817

zIm R

1,R

-1,1

Figure L.13. Contour used for solving the integral in Example L.7.

-1

1,-1

1

R,1

We can then specify the branch cut by fixing the ranges on θa and θb to be 0 < θa < 2π and

0 < θb < 2π

Aside from being branch points, the points z = ±1 are also singular points. We can then choose the contour shown in Figure L.13 and implement the method of residues. The closed-contour C is given by C

R + R,1 + (1)lower + 1,−1 + (−1) + −1,1 + (1)upper + 1,R   R + (1) + (−1) + (R,1 + 1,R ) + (1,−1 + −1,1 )

= =

Following earlier methods, we can evaluate the integrals along the three circular paths: the outer circle R and the pair of inner circles (1) (−1) , to yield zero values as the limits of R → ∞ and  → 0 are approached, respectively. Thus we need to evaluate only the four remaining straight paths. Because f (z) is multivalued, the path along a common segment, but in opposite directions, may not necessarily cancel. We now show that the integrals along 1,R and R,1 will cancel, whereas the integrals along 1,−1 and −1,1 will not. Along the path R,1 , we have zim = 0, 1 < zre ≤ R, θa = 2π and θb = 2π, thus  f (z)R,1 =

1 1 = √ √ (1 + x2 ) ra rb e2πi (1 + x2 ) x2 − 1

Similarly, along path 1,R , we have zim = 0, 1 < zre < R, θa = 0 and θb = 0,  f (z)1,R =

1 1 = √ √ 2 (1 + x2 ) ra rb (1 + x ) x2 − 1

The sum of integrals along both 1,R and R,1 is then given by 

 1,R

f (z)dz +

 R,1

f (z)dz

=

=

R

1 dx √ 1 (1 + x2 ) x2 − 1  1 1 + dx √ 2 R (1 + x ) x2 − 1

0

Along the path 1,−1 , we have zim = 0, −1 < zre ≤ 1, θa = π and θb = 2π, thus  f (z)1,−1 =

1 −1 = √ √ 2 (1 + x2 ) ra rb e3πi/2 (1 + x )i 1 − x2

zRe

818

Appendix L: Additional Details and Fortification for Chapter 12

Similarly, along path −1,1 , we have zim = 0, −1 < zre < 1, θa = π and θb = 0,  f (z)−1,1 =

(1 +

x2 )

1 1 = √ √ πi/2 ra rb e (1 + x2 )i 1 − x2

Note that we used ra rb = (1 − x2 ) because |x| < 1. Thus the sum of integrals along both 1,−1 and −1,1 is given by 

 1,−1

f (z)dz +

 −1,1

−1

=

f (z)dz

1

+ 2 i

=



(1 +  1 −1 1

−1

−1 dx √ 1 − x2

x2 )i

(1 +

1 dx √ 1 − x2

x2 )i

1 dx √ (1 + x2 ) 1 − x2

Next, we need to calculate the residues at the poles z = ±i. Note that because the function is multivalued, we need to be careful when taking the limits of the square root. First, consider the pole z = −i. At this point, we have z − 1 = −i − 1

=

z + 1 = −1 + i

=

√ √

2 e5πi/4 2 e7πi/4

Thus 

Res−i [ f ]

= = =

z+ i ! lim z→−i (1 + z2 ) (z − 1)(z + 1)    1 1 √ −2i 2 e3πi/2



−1 √ 2 2

For the other pole, z = i, z− 1 = i − 1

=

z+ 1 = i + 1

=

√ 3πi/4 2e √ πi/4 2e

and 

Resi [ f ]

= = =

z− i ! lim 2 z→i (1 + z ) (z − 1)(z + 1)    1 1 √ 2i 2 eπi/2 −1 √ 2 2



Appendix L: Additional Details and Fortification for Chapter 12

Finally, we combine all the previous calculations to obtain 

2 i



1

−1



f (z)dz

=

2πi (Res−i [ f ] + Resi [ f ])

1 dx √ (1 + x2 ) x2 − 1

=

  −1 2πi √ 2

1 dx √ x2 − 1

=

π √ 2

C

1

−1

(1 +

x2 )

L.3 Dirichlet Conditions and the Fourier Integral Theorem Definition L.5. A function f (x) is said to satisfy the Dirichlet conditions in the interval (a, b), if the interval (a, b) can be partitioned into a finite number of subintervals such that f (x) is bounded and monotonic in each of these subintervals. This means: 1. There are a finite number of maxima and minima for f (x) in (a, b). 2. f (x) has no infinite discontinuities, but it can have a finite number of bounded discontinuities. Then we have the following theorem, known as the Fourier integral theorem. ∞ Let f (x) be such that −∞ | f (x)|dx < ∞, and let f (x) satisfy Dirichlet’s conditions given in definition L.5 for (a, b) = (−∞, ∞), then    1 1 ∞ ∞ f (x+ ) + f (x− ) = f (t) cos (ω(x − t)) dt dω (L.54) 2 π 0 −∞

THEOREM L.8.

where f (x+ ) = lim f (x + |η|) η→0

and

f (x− ) = lim f (x − |η|) η→0

As opposed to the prior approach of taking limits on the Fourier series (cf. (12.5)), equation (L.54) given in Theorem L.8 can be more correctly derived from another important result known as Dirichlet’s integral theorem,   1 ∞ 1 sin (θη) + − f (x ) + f (x ) = lim f (x + η) dη (L.55) θ→∞ π −∞ 2 η

PROOF.

as long as f (x) satisfy Dirichlet’s conditions. The proof of (L.55) is given in section L.6.1 (page 836). Let t = x + η. Also, we use the fact that  θ sin (θη) = cos (ηω) dω (L.56) η 0 Substituting (L.56) into (L.55) with x held fixed, we get  θ   1 1 ∞ + − f (x ) + f (x ) = lim f (t) cos ((x − t)ω) dω dt θ→∞ π −∞ 2 0

(L.57)

819

820

Appendix L: Additional Details and Fortification for Chapter 12

The last important detail deals with the validity of interchanging  ∞the sequence of integration in (L.57). With the assumption in Theorem L.8 that ( −∞ | f (t)|dt < ∞), we can show that (see Section L.6.2),  lim



∞

θ→∞ −∞

f (t)

θ

 θ

cos ((x − t)ω) dω dt = lim

θ→∞ 0

0

∞ −∞

f (t) cos ((x − t)ω) dt dω (L.58)

So with (L.58) substituted to (L.57), we obtain the Fourier integral equation given in (L.54)

L.4 Brief Introduction to Distribution Theory and Delta Distributions In this appendix, we introduce some of the basic theory and tools to generalize the concept of functions, with special attention to the construction of delta distributions. We also include a brief discussion of a very important class of distributions, called tempered distributions, that generalizes the theory of Fourier transforms for functions that may not be absolutely integrable.

L.4.1 The Delta Distribution (Delta Function) The delta distribution, denoted by δ (t) and often known as the delta function, is an important operation in applied mathematics. However, it does not satisfy the classical requirements of functions; for example, it is not defined at t = 0. Instead, a new concept known as distributions (also known as generalized functions) had to be constructed to give the necessary mathematical rigor to δ (t). Once the theory for distribution was built, the constructs allow for the definition of other distributions, including the derivatives of δ (t) and δ (g(t)), where g(t) is a continuous function. Consider the Heaviside step function, H (t), defined as + H (t) =

0 1

if if

t 0. However, if ϑ(t) is strictly monotonically decreasing, z → −∞ as t → ∞ and dϑ/dt > 0. For the latter case, the lower limit of integration will be +∞ and the upper limit is −∞. Thus, for either case, by fixing the upper limit to be +∞ and the lower limit to be −∞, we take the absolute value of dϑ/dt when defining ζ(t) in (L.70). 3. Derivatives of Distributions. The derivative of distribution Dist (t), denoted by Dist (t), is also a distribution. After applying integration by parts, the operation of Dist (t) is given by R S F E d  Dist (t) , ϕ(t) Dist (t) , ϕ(t) = dt  ∞ dDist (t) = ϕ(t)dt dt −∞  ∞ dϕ = − Dist (t) dt dt −∞ R S dϕ(t) = − Dist (t) , dt (L.75) dt Using the preceding operations of distributions, we have the following theorem that describes the calculus available for distributions. Let Dist (t), Dist1 (t), and Dist2 (t) be distributions, g(t) be a C ∞ function, and α be a constant, then

THEOREM L.9.

1. The derivative of sums of distributions are given by d d d (Dist1 (t) + Dist2 (t)) = (Dist1 (t)) + (Dist2 (t)) dt dt dt

(L.76)

2. The derivative of a scalar product of a distribution with g(t) is given by d d dg Dist (t) + g(t) Dist (t) [g(t)Dist (t)] = dt dt dt

(L.77)

Appendix L: Additional Details and Fortification for Chapter 12

For the special case of g(t) = α, d d [αDist (t)] = α Dist (t) dt dt

(L.78)

3. The derivative of a distribution under argument transformation ϑ(t), where ϑ(t) is an invertible monotonic function, is given by ' ( d dϑ d (L.79) [Dist (ϑ(t))] = [Dist (ϑ)] dt dt dϑ PROOF.

See Section L.8.

L.4.3 Properties and Identities of Delta Distribution As a consequence of distribution theory, some of the properties and identities of delta distribution are given by: 1. Sifting property.



∞ −∞

 δ (t − α) f (t)dt

= =

2. Rescaling property. Let α = 0,  ∞ δ (αt) f (t)dt −∞

∞

−∞

δ (t) f (t + α)dt

f (α) 

∞

=

1 |α|

=

1 f (0) |α|

−∞

(L.80)

δ (t) f (t/α)dt (L.81)

A special case is when α = −1, then δ (−t) = δ (t). 3. Identities Involving Derivatives. N O R n S (n−k) d dk k d δ (t) , f (t) = (−1) δ (t) , k f (t) , 0 ≤ k ≤ n dtn dt dt(n−k)

tn

⎧ ⎨

0 d δ (t) = m ⎩ (−1)n m! d(m−n) δ (t) dt (m−n)! dt(m−n) m

if

0≤m q1 , 

q2

lim

q1 →∞ q 1

sin(q) dq = 0 q

(L.135)

Assume that f (x + η) is monotonic in a subinterval (α, β) of (a, b) for the case a > 0, the mean value theorem says there exists a < ξ < b such that, with q = θη, 

β

α

sin(θη) f (x + η) dη η



+

=

f (x + α ) f (x + α+ )

=

ξ

sin(θη) dη + f (x + β− ) η

α



θξ

sin(q) dq + f (x + β− ) q

θα

 

β

sin(θη) dη η

θβ

sin(q) dq q

ξ

θξ

and with (L.135), 

β

lim

θ→∞ α

f (x + η)

sin(θη) dη = 0 η

(L.136)

Note that so far, (L.136) has been shown to apply to a subinterval where f (x + η) is monotonic. However, because f (x + η) satisfies Dirichlet’s conditions, the interval (a, b) can be partitioned into n subintervals (ai , ai+1 ), with 0 < a = a0 < a1 < · · · < an = b such that f (x) is monotonic inside each subinterval (e.g., with ai occurring either at a discontinuity, minima, or maxima of f (x)). Thus  lim

b

θ→∞ (a>0)

 sin(θη) dη = lim θ→∞ η n−1

f (x + η)



i=0

ai+1

ai

f (x + η)

sin(θη) dη = 0 η

(L.137)

Similarly with b < 0, the same approach can be used to show 

(b0) sin(θη) lim f (x + η) dη = f (x+ ) + f (x− ) (L.141) θ→∞ π (a 0 and τ > T ,    



∞

τ

θ

f (t) 0

  cos ((x − t)ω) dω dt

 ∞   sin((x − t)ω)   f (t) dt   x−t τ  ∞   sin(ηω)   f (x + η) dη   η x+τ    1  ∞ <  | f (x + η)| dη   2τθ τ τ

= =
0 then R

t tn

dn δ (t) = (−1)n n!δ (t) dtn

dn δ (t) , ϕ(t) dtn

S =

E F (−1)n n! δ (t) , t ϕ(t)

=

0

Thus tn

dm δ (t) = 0 dtm

if

0≤m 0 is small enough such that g(t) is monotonic and invertible in the range (rk − ) ≤ t ≤ (rk + ) for all k.

Appendix L: Additional Details and Fortification for Chapter 12

We can now apply an equation similar to (L.69) for each integral term,    g(rk +)  rk + ϕ g −1 (z) δ (g(t)) ϕ(t)dt = δ (z) dz |dg/dt|(g −1 (z)) g(rk −) rk − =

ϕ (rk ) |dg/dt|t=rk

=

E F 1 δ (t − rk ) , ϕ(t) |dg/dt|t=rk

Combining both results, E F δ (g(t)) , ϕ(t)

=

N  k=1

N =

E F 1 δ (t − rk ) , ϕ(t) |dg/dt|t=rk

N '  k=1

PROOF OF THEOREM L.10.

O ( 1 δ (t − rk ) , ϕ(t) |dg/dt|t=rk

Using (L.72),

R  S E F t F (α, t), ϕ(t) = α f (t), ϕ α

then E F F (α, t) − δ (t) , ϕ(t)

R

=

 S t f (t), ϕ − ϕ(0) α    ∞ then with f (t)dt = 1 

=

−∞

∞

−∞

  t f (t) dt α

where (t) = ϕ(t) − ϕ(0) Taking absolute values of both sides, we obtain the following inequality, E F  F (α, t) − δ (t) , ϕ(t)  ≤ A + B where, A

=

B

=

 −q       ∞   t t  dt + f (t) dt f (t)  α α −∞ q  q      t  dt f (t)  α −q

Now choose κ > 0, q > 0 and α > q/κ such that 1. |(t)|  −q < ε for |t|  ∞< κ 2. −∞ | f (t)|dt + q | f (t)|dt < ε

849

850

Appendix L: Additional Details and Fortification for Chapter 12

then

or

A

≤

B

≤

2ε · max |ϕ(t)| t   ∞    f (t)dt ε· −∞

  E  F  F (α, t) − δ (t) , ϕ(t)  ≤ ε 2 max |ϕ(t)| +   t

∞ −∞

  f (t)dt

Because all the terms on the right-hand side of the inequality is fixed except for ε, we can choose ε arbitrarily small. Hence lim F (α, t) = δ (t)

α→∞

APPENDIX M

Additional Details and Fortification for Chapter 13

M.1 Method of Undetermined Coefficients for Finite Difference Approximation of Mixed Partial Derivative For the case of mixed partial derivatives, we use the general formula of the Taylor series of u(x, t) expanded around (xk , tq ): (q+i) uk+j

=

∞  ∞ 

m,i f m,  γ,j

(M.1)

m=0 =0

where f m,

∂ m+ = m  ∂t ∂x

  u

(xk ,tq )

tm x

and

m,i  γ,j = γ,j γm,i

and γ,j has been defined in (13.11). The computation for the general case involves high-order tensorial sums. A ∂2u simple example is the approximation of . ∂x∂t Approximation of Mixed Partial Derivatives. Let D1,x,1,y be the approximation of the mixed derivative defined as a linear combination of the values at neighboring points,  1 1 1   ∂ 2 u  D1,x,1,y = uk+i,n+j αi,j = + Error (M.2)

x y ∂x∂y (xk ,yn )

EXAMPLE M.1.

i=−1 j =−1

Applying (M.1), we obtain ⎞ ⎛ 2  2 1 1    m,i ⎝ f m,  γ,j αi,j ⎠ − f 1,1 = t x (Error) =0 m=0

i=−1 j =−1

⎛ −⎝ ⎛ −⎝

∞  ∞ 

f m,

1 1  

=3 m=0

i=−1 j =−1

∞  ∞ 

1 1  

=0 m=3

f m,

⎞ m,i  γ,j αi,j ⎠

⎞ m,i  γ,j αi,j ⎠

i=−1 j =−1

(M.3) 851

852

Appendix M: Additional Details and Fortification for Chapter 13

Setting the left-hand side (M.3) equal to 0 results in a set of nine independent linear equations: + 0 if (i, j ) = (1, 1) m,i αi,j =  γ,j 1 if (i, j ) = (1, 1) Solving these equations, we obtain ij i, j = −1, 0, 1 4 which yields the following finite difference approximation of the mixed partial derivative:    ∂ 2 u  1 (k+1) (k+1) (k−1) (k−1) ≈ u (M.4) − u − u + u n−1 n+1 n−1 ∂x∂y  4 x y n+1 αi,j =

(xk ,yn )

To determine the order of truncation error, note that the coefficients of the lower order terms of f m, are ⎧ ⎪ ⎪ 0 if l = 0, or m = 0 ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ 1   1  1 ⎨  m,i 1 + (−1)m+1 if l = 1  γ,j αi,j = 2m! ⎪ ⎪ i=−1 j =−1 ⎪  ⎪ ⎪ 1  ⎪ +1 ⎪ 1 + (−1) if m = 1 ⎪ ⎩ 2! yielding

 4   4   

t2 ∂

x2 ∂   Error = u + u + ···   3! ∂t3 ∂x1 3! ∂t1 ∂x3 (x,t) (x,t)   or Error = O t2 , x2 .

M.2 Finite Difference Formulas for 3D Cases For the 3D, time-invariant case, a general second-order linear differential equation is given by

μxx (x, y, z)

∂2u ∂2u ∂2u + μ (x, y, z) + μ (x, y, z) yy zz ∂x2 ∂y2 ∂z2

+ μxy (x, y, z)

∂2u ∂2u ∂2u + μyz(x, y, z) + μxz(x, y, z) ∂x∂y ∂y∂z ∂x∂z

+ βx (x, y, z)

∂u ∂u ∂u + βy (x, y, z) + βz(x, y, z) ∂x ∂y ∂z + ζ(x, y, z)u + η(x, y, z)

=

0

(M.5)

Let the superscript “(3()” denote matrix augmentation that will flatten the 3D tensor into a matrix representation. For instance, for k = 1, . . . , K, n = 1, . . . , N, and

Appendix M: Additional Details and Fortification for Chapter 13

m = 1, . . . , M, we have the following K × NM matrix for uknm : ⎛ u1,1,1 · · · u1,N,1 u1,1,M · · · ⎜ . . .. (3() . .. .. .. .. U = ⎝ . ··· . uK,1,1

···

uK,N,1

uK,1,M

ζ1,1,1 ⎜ .. ⎝ . ζK,1,1

··· .. . ···

ζ1,N,1 .. .

ζ1,1,M .. .

⎛

ζ(3()

=

···

ζK,N,1

ζK,1,M

⎞ u1,N,M ⎟ .. ⎠ . uK,N,M

···

··· .. . ···

⎞ ζ1,N,M ⎟ .. ⎠ . ζK,N,M

etc. where uk,n,m = u(k x, n y, m z), ζk,n,m = ζ(k x, n y, m z), etc. Likewise, let the superscripts “(2(, x)” and “(2(, y)” denote column augmentation, as the indices are incremented along the x and y directions, respectively. For instance,     (2(,x) b(1,z) = b(1,z) k=1 · · · b(1,z) k=K The partial derivatives can then be approximated by finite difference approximations in matrix forms as follows: ∂u (3() → D(1,x) U (3() + B(1,x) ∂x    (3() ∂u T T → U (3() IM ⊗ D(1,y) + B(1,y) ∂y   ∂u T (1,z) → U (3() D(1,z) ⊗ IN + B ∂z

; ; ;

∂2u (3() → D(2,x) U (3() + B(2,x) ∂x2    (3() ∂2u (3() T T I + B → U ⊗ D M (2,y) (2,y) ∂y2   ∂2u (3() T (2,z) D +B → U ⊗ I N (2,z) ∂z2

∂2u ∂x∂y

→

  (3() T D(1,x) U (3() IM ⊗ D(1,y) + B(1,x,1,y)

∂2u ∂y∂z

→

  T T (1,y,1,z) +B ⊗ D(1,y) U (3() D(1,z)

∂2u ∂x∂z

→

  T (1,x,1,z) D(1,x) U (3() D(1,z) ⊗ IN + B

where D(1,x) , D(1,y) , D(1,z) , D(2,x) , D(2,y) , and D(2,z) are matrices that can take forms such as those given in Section 13.2.2 depending on order of approximation and boundary conditions. The matrices B(1,x) , B(2,x) , . . . , and so forth contain the boundary data. 2,z are given by a sequence of transformations as 1,z and B The new matrices B '   (2(,y) (T (2(,x) (1,z) = reshape b B , K, NM (M.6) (1,z)

(2,z) B

=

'  (T   (2(,x) (2(,y) b(2,z) reshape , K, NM

1,x,1,z and B 1,y,1,z are left as exercises.) (The matrices B

(M.7)

853

854

Appendix M: Additional Details and Fortification for Chapter 13

Just as in the previous section, we can use the properties of matrix vectorizations to obtain the following linear equation problem corresponding to (M.5):   R3D vec U (3() = f3D

(M.8)

where, R3D

=

' (dv  dv μxx (3() IM ⊗ IN ⊗ D(2,x) + μyy (3() IM ⊗ D(2,y) ⊗ IK +

 dv μzz(3() D(2,z) ⊗ IN ⊗ IK

' ' (dv (dv (3() (3() + μxy IM ⊗ D(1,y) ⊗ D(1,x) + μyz D(1,z) ⊗ D(1,y) ⊗ IK +

 dv μxz(3() D(1,z) ⊗ IN ⊗ D(1,x)

' (dv  dv (3() (3() + βx IM ⊗ IN ⊗ D(1,x) + βy IM ⊗ D(1,y) ⊗ IK + +

f3D

=

 dv βz(3() D(1,z) ⊗ IN ⊗ IK

 dv ζ(3()

' (dv  dv    (3()  (3() μxx (3() vec B(2,x) + μyy (3() vec B(2,y)T +

 dv   (2,z) μzz(3() vec B

(dv ' ' (dv     (3() (3() (3() (1,y,1,z) + μxy vec B(1,x,1,y) + μyz vec B +

 dv   (1,x,1,z) μxz(3() vec B

+

' (dv   dv   (3()  (3() T βx (3() vec B(1,x) + βy (3() vec B(1,y) +

 dv   (1,z) βz(3() vec B

  + vec η(3()

EXAMPLE M.2.

Consider the 3D Poisson equation ∇ 2 u = η(x, y, z)

0 ≤ x, y, z ≤ 1

(M.9)

Appendix M: Additional Details and Fortification for Chapter 13

where,



(2  4 η(x, y, z) = exp −2 [z − x] − 5 1 − z − y 5 ⎡ ⎤  2  2 4 52 ⎢ 122 ⎥ × ⎣− + 16 (x − z)2 + 100 1 − z − y + 8 − z − 8y + 4x ⎦ 5 5 5

'

2

subject to the following six Dirichlet boundary conditions:  ' (2  4 u (0, y, z) = α0 (y, z) = exp −2z2 − 5 1 − z − y 5  ' (2  4 2 u (1, y, z) = α1 (y, z) = exp −2 [z − 1] − 5 1 − z − y 5  ' ( 4 2 2 u (x, 0, z) = β0 (x, z) = exp −2 [z − x] − 5 1 − z 5   16 u (x, 1, z) = β1 (x, z) = exp −2 [z − x]2 − z2 5   u (x, y, 0) = γ0 (x, y) = exp −2x2 − 5 [1 − y]2  ' (2  1 2 u (x, y, 1) = γ1 (x, y) = exp −2 [1 − x] − 5 −y 5 The exact solution is given by



(2  4 u (x, y, z) = exp −2 [z − x] − 5 1 − z − y 5

(M.10)

'

2

(M.11)

Using x = y = z = 0.05, and central difference formulas for D(2,x) , D(2,y) ,  B(2,x) D(2,y)  , B(2,y) , and B(2,z) , the linear equation (M.8) can be solved for  , (3() . The results are shown in Figure M.1 at different values of z, where vec U the approximations are shown as points, whereas the exact solutions are shown as surface plots. (A MATLAB file poisson_3d.m is available on the book’s webpage that implements the finite difference solution and obtains the plots shown in this example.) The errors from the exact solution (M.11) are shown in Figure M.2 at different fixed values of z. The errors are in the range ±1.7 × 10−3 .

M.3 Finite Difference Solutions of Linear Hyperbolic Equations Consider the following linear hyperbolic equations ∂ ∂  u+A  u = c ∂t ∂x

(M.12)

855

856

Appendix M: Additional Details and Fortification for Chapter 13

z= 0.1

z= 0.2

1

u 0 1

y

1 0 0

z= 0.3

1

1

0.5

0.5

0 1

0.5

x

0.5

0 0

z= 0.4

1

0 1

0.5

0 0

z= 0.5 1

1

0.5

0.5

0.5

0.5

0 0

0.5

1

0 1

0.5

0.5

0 0

z= 0.7

1

0 1

0.5

0 0

z= 0.8 1

1

0.5

0.5

0.5

0.5

0 0

0.5

1

0 1

0.5

0 0

0.5

1

z= 0.9

1

0 1

1

z= 0.6

1

0 1

0.5

0.5

1

0 1

0.5

0 0

0.5

1

Figure M.1. The finite difference solution to (M.9) at different values of z, subject to conditions (M.10). The approximations are shown as points, whereas the exact solutions, (M.11), at the corresponding z values are shown as surface plots.

where  u = ( u1 , . . . , uJ )T and A is a constant J × J matrix. If A is diagonalizable, that is, there exist a nonsingular matrix V and a diagonal matrix  such that A = VV −1 , then with u u = V −1

c = V −1 c

we can decouple (M.12) into J equations ∂ui ∂ui + λi = ci ∂t ∂x Thus in the discussion that follows, we consider ∂u ∂u +λ =c ∂t ∂x

(M.13)

as a representative system for handling a system of first-order hyperbolic equations. However, in our discussion of the scalar case, we allow for c = c(x, t).

M.3.1 Upwind Schemes We can use either forward, backward, or central difference approximations for ∂u/∂x toward a semi-discrete approach. Time marching can then be implemented by a forward Euler or backward Euler. This will yield six types of schemes, namely

Appendix M: Additional Details and Fortification for Chapter 13 −3

x 10

−3

x 10

z= 0.1

2

u −2 1

y 0 0

x

1

−3

x 10

z= 0.2

2

2

0

0

−2 1

0.5

0 0

−3

0.5

1

0.5

0 0

z= 0.5

z= 0.6

2

2

0

0

0

0.5

0 0

0.5

1

−2 1

−3

0.5

0 0

0.5

1

−2 1

−3

x 10

0

0

0

−2 1

−2 1

1

1

2

−2 1

0.5

0.5

z= 0.9

2

0 0

0 0

x 10

z= 0.8

2

0.5

0.5 −3

x 10

z= 0.7

1

x 10

2

−2 1

0.5

−3

x 10

z= 0.4

z= 0.3

−2 1

−3

x 10

857

0.5

0 0

0.5

1

0.5

0 0

0.5

1

Figure M.2. The error distribution between the finite difference approximation (using central difference formulas) and the exact solutions, (M.11), at different z values.

forward-time-forward-space (FTFS), forward-time-central-space (FTCS), forwardtime-backward-space (FTBS), backward-time-forward-space (BTFS), backwardtime-central-space (BTCS), and backward-time-backward-space (BTBS). Each scheme will have different stability ranges for t in relation to x and λ. In Table M.1, we summarize the different upwind schemes and their stability based on another parameter η η=λ

t

x

(M.14)

which is known as the Courant number. The stability conditions included in the table are obtained using the von Neumann method and are given as Exercise in E13.15. We can make the following observations: 1. The forward-time schemes: FTFS, FTCS, and FTBS are explicit schemes, whereas the backward-time schemes: BTFS, BTCS and BTBS are implicit schemes. 2. The central-space schemes are given by FTCS and BTCS, with the explicit FTCS being unstable and the implicit BTCS being unconditionally stable. 3. The noncentral space schemes have their stability dependent on the sign of η, or equivalently on the sign of λ. Both forward-space schemes, FTFS and BTFS, are

858

Appendix M: Additional Details and Fortification for Chapter 13 Table M.1. Basic finite difference schemes for scalar hyperbolic equations Scheme

Approximation equation

FTFS

uk

FTCS

uk

FTBS

uk

BTFS

(1 − η) uk

BTCS

uk

BTBS

(1 + η) uk

Leapfrog

uk

Lax-Friedrichs

uk

(q)

(q+1)

= (1 + η) uk − ηuk+1 + tck

(q+1)

= uk −

(q+1)

= ηuk−1 + (1 − η) uk + tck

(q)

(q+1)

(q+1)

+

(q+1)

= uk

(q+1)

=

(q+1)

(q)

−1 ≤ η ≤ 0

 η  (q) (q) (q) uk+1 − uk−1 + tck 2 (q)

(q+1)

None

(q)

(q+1)

+ ηuk+1 + tck

0≤η≤1 (q)

= uk

η≤0

 η  (q+1) (q+1) (q+1) (q) uk+1 − uk−1 + tck = uk 2 (q+1)

uk

(q)

(q)

(q+1)

(q+1)

− ηuk−1 + tck

(q−1)



(q)

(q)

= uk

(q)

=

(q)

|η| ≤ 1

− ηun+1 + ηun−1 + 2 tck

  (q) 1 − η2 uk +

1 2

All

η≥0

   1−η 1+η (q) (q) (q) uk+1 + uk−1 + tck 2 2

|η| ≤ 1

 (q)  2 η − η uk+1

  (q) + 21 η2 + η uk−1   ∂c ∂c (q) 2 (q) + ck t + −λ

t ∂t ∂x k

Lax-Wendroff

Crank-Nicholson

Stability region

η (q+1) η (q+1) (q+1) + uk − uk−1 = u 4 k+1 4 η (q) η (q) (q) (q+1/2) − uk+1 + uk + uk−1 + tck 4 4

|η| ≤ 1

All

stable only for negative η values, whereas both backward-space schemes, FTBS and BTBS, are stable only for positive η values.1 From the last observation, we can still recover the use of noncentral schemes by switching between forward-space and backward-space schemes depending on the sign of λ. This combination is called the upwind schemes, because the direction of space difference is adjusted to be opposite to the wave speed λ. Specifically, with η(+) = 1

η + |η| 2

and

η(−) =

η − |η| 2

(M.15)

Note that even though BTFS and BTBS are both implicit schemes, neither are unconditionally stable.

Appendix M: Additional Details and Fortification for Chapter 13

we have the explicit upwind scheme, which combines both FTFS and FTBS in one equation,   (q+1) (q) (q) (q) = 1 + η(−) − η(+) uk + η(+) uk−1 − η(−) uk+1 (M.16) uk and whose stability range is given by 0 < |η| < 1. Likewise, we have the implicit upwind scheme, which combines both BTFS and BTBS in one equation,   (q+1) (q+1) (q+1) (q) 1 − η(−) + η(+) uk − η(+) uk−1 + η(−) uk+1 = uk (M.17) whose stability range is given by 0 < |η|.

M.3.2 Other Finite Difference Schemes There are four more important schemes: the leapfrog (or CTCS) scheme, the LaxFriedrichs scheme, the Lax-Wendroff scheme, and Crank-Nicholson scheme. The first three are explicit, whereas the last one is an implicit scheme. The leapfrog and the Lax-Friedrichs schemes are improvements to the FTCS scheme to overcome its unconditional instability. The leapfrog scheme uses the central difference approximation for ∂u/∂t. Thus we have (q)

λ

(q)

uk+1 − uk−1 2 x

(q+1)

+

uk

(q−1)

− uk 2 t

(q)

= ck

(M.18)

Note that the leapfrog scheme needs values at both tq and tq−1 to obtain values at tq+1 . Thus the leapfrog schemes often require other one-step marching, such as Lax-Friedrich or Lax-Wendroff to provide it with values at t1 , and then continue with the leapfrog for tq , q ≥ 2. The Lax-Friedrichs scheme approximates the time derivative as a forward time  (q+1) (q) (q) difference, but between uk and the average at the current point, 21 uk+1 + uk−1 . Thus the scheme is given by   (q+1) (q) (q) (q) (q) uk − 21 uk+1 + uk−1 uk+1 − uk−1 (q) λ + = ck (M.19) 2 x

t Note that the leapfrog scheme used the values at tq−1 , whereas the Lax-Friedrichs continues to stay within tq . The third explicit finite difference scheme uses the Taylor series approximation for u,     ∂u  1 ∂ 2 u  (q+1) (q) = uk +

t + uk

t2 + O t3 (M.20)   2 ∂t t=q t,x=k x 2 ∂t t=q t,x=k x and then substitutes the following identities obtained from the given differential equation ∂u ∂u = −λ + c and ∂t ∂x

2 ∂2u ∂c 2∂ u = λ − λc + ∂t2 ∂x2 ∂t

into (M.20). Afterward, the central approximation is used for ∂u/∂x and   difference ∂ 2 u/∂x2 . After truncation of O t3 terms, the following scheme, known as the

859

860

Appendix M: Additional Details and Fortification for Chapter 13

Lax-Wendroff scheme, results (q+1)

uk

 1 t2  

t  (q) (q) (q) (q) (q) uk+1 − uk−1 + λ2 2 uk+1 − 2uk + uk−1 2 x 2 x  (q) ∂c ∂c (q) + ck t + −λ

t2 ∂t ∂x k

(q)

=

uk − λ

or (q+1)

=

uk

 (q) 1  2   (q)  (q) 1 2 (q) 1 − η2 uk + η − η uk+1 + η + η uk−1 + ck t 2 2   ∂c ∂c (q) 2 + −λ

t (M.21) ∂t ∂x k

Using the von Neumann method, one can show that the stability range of the three explicit schemes, namely the leapfrog, Lax-Friedrichs, and Lax-Wendroff schemes, given in (M.18), (M.19), and (M.21), respectively,  given by |η| ≤ 1.  2 are 2all , O ( x, t), and ,

t The approximation errors for these methods are O

x   2 2 O x , t for leapfrog, Lax-Friedrichs, and Lax-Wendroff schemes, respectively. The Crank-Nicholson scheme is an implicit scheme that could be seen as an attempt to improve the accuracy of the BTCS scheme,  may be uncondition which ally stable but only has approximation errors of O x2 , t . However, unlike the leapfrog scheme, where values at tq−1 are introduced, this method tries to avoid this from occurring by using a central difference approximation at a point between tq+1 and tq , that is, at t = tq+1/2 , with a time increment t/2. However, by doing so, the spatial derivative at t = tq+1/2 must be estimated by averages. Thus the CrankNicholson scheme uses the following approximation for the time derivative: λ 2 x

8

(q+1)

(q)

uk+1 + uk+1 2

(q+1)

−

(q)

uk−1 + uk−1 2

9

 +

(q+1)

uk

(q)

− uk 2( t/2)

 (q+1/2)

= ck

or η (q+1) η (q+1) η (q) η (q) (q+1) (q) (q+1/2) uk+1 + uk − uk−1 = − uk+1 + uk + uk−1 + tck (M.22) 4 4 4 4   The approximation error of the Crank-Nicholson scheme is O x2 , t2 . Using the von Neumann method, we can show that the Crank-Nicholson scheme, like the BTCS scheme, is unconditionally stable. EXAMPLE M.3.

For the scalar hyperbolic partial differential equation given by

∂u ∂u + 0.5 =0 (M.23) ∂t ∂x we consider both a continuous initial condition and a discontinuous initial condition. 1. Continuous initial condition. Let initial condition be a Gaussian function given by u(0, t) = e−8(5x−1)

2

(M.24)

Appendix M: Additional Details and Fortification for Chapter 13

t 0 0

x BTCS

t0 x 0 Lax-Wendroff

u x

x

Using the various stable schemes, the finite-difference solutions with

x = t = 0.01 are shown in Figure M.3. It appears that the leapfrog, LaxWendroff, and Crank-Nicholson schemes yielded good approximations. 2. Discontinuous initial condition. Let the initial condition be a square pulse given by + 1 if 0.2 ≤ x ≤ 0.4 u(0, t) = (M.25) 0 otherwise

UPWIND (Implicit)

1

0.5

0.5

u

u

UPWIND (Explicit)

1

0

0

0.5 x Leapfrog

1

1

1

0.5

0.5

u

u

0

0

0.5 x Lax−Wendroff

1

0

0.5 x Crank−Nicholson

1

0

0.5 x

1

0

0

0.5 x BTCS

1

1

1

0.5

0.5

u

u

0

0

0

0

0.5 x

1

1

1 0 1 t 0 0

1

1

1 0 1 t0 x 0 Crank-Nicholoson

1

1 0 1 t 0 0

1 0 1

1

1 0 1 t 0 0

u

x Leapfrog

u

u

Figure M.3. Numerical solutions for continuous initial condition using the various schemes.

UPWIND (Implicit)

1 0 1

u

u

UPWIND (Explicit)

861

Figure M.4. Comparison with exact solutions for different schemes at t = 1. The exact solution is given as dashed lines.

1

862

Appendix M: Additional Details and Fortification for Chapter 13

x Leapfrog

1 0 1 t0 x 0 Lax-Wendroff

1

1 0 1 t 0 0

x BTCS

u

t 0 0

x

1

Figure M.5. Numerical solutions for discontinuous initial condition using the various schemes.

t0 0 x Crank-Nicholson

1

1 0 1

1

1 0 1

u

u

t 0 0

u

UPWIND (Implicit) u

u

UPWIND (Explicit) 1 0 1

1

1 0 1 t 0 0

x

1

Using the various stable schemes, the finite-difference solutions with

x = t = 0.01 are shown in Figure M.5. As one can observe from the plots, none of the schemes match the exact solution very well. This is due to numerical dissipation introduce by the schemes. Dissipation was instrumental for stability, but it also smoothed the discontinuity. However, the other schemes had growing amount of oscillations. These are due to the spurious roots of the schemes. Significant amounts of oscillation throughout the spatial domain can be observed in both the leapfrog and Crank-Nicholson schemes. The Lax-Wendroff appears to perform the best; however, a smaller mesh size should improve the approximations. More importantly, however, is that if one had chosen |η| = 1, both the Lax-Wendroff and Lax-Friedrich schemes reduce to yield an exact solution as shown in Figure M.7 because the discontinuity   will travel along the char1 acteristic; that is, with c(x, t) = 0 and t = x   (or |η| = 1), both schemes λ reduce to ⎧ ⎪ (q) ⎪ ⎨ uk+1 if η = −1 (q+1) uk = (q) ⎪ ⎪ ⎩ uk−1 if η = +1

The example shows that the Lax-Wendroff performed quite well, especially when t was chosen carefully so that |η| = 1. Note that the case in which it yielded an exact solution (at the grid points) is limited primarily to a constant η and zero homogeneous case, that is, c(x, t) = 0. The other issue remains that Lax-Wendroff and Lax-Friedrich are still explicit time-marching methods.

Appendix M: Additional Details and Fortification for Chapter 13 UPWIND (Implicit)

1

1

0.5

0.5

u

u

UPWIND (Explicit)

0

0

0.5 x Leapfrog

1

1

1

0.5

0.5

u

u

0

0

0

0.5 x Lax−Wendroff

1

0

0.5 x Crank−Nicholson

1

0

0.5 x

1

0

0

0.5 x BTCS

1

1

1

0.5

0.5

u

u

863

0

0

0

0.5 x

1

Figure M.6. Comparison with exact solutions for different schemes at t = 1.

M.4 Alternating Direction Implicit (ADI) Schemes Let matrix G be a multidiagonal, banded matrix of width ω, that is, Gij = 0 for |i − j | > ω. In general, the LU factorization of G will result in L and U matrices that are banded with the same width. Unfortunately, the matrices generated during finite-difference methods of two or three spatial-dimensional systems are likely to have very wide bands, even though the matrices are very sparse. For instance, matrix R in Example 13.9 will have a band of width N. Yet in any row of R, there are only at most five-nonzero entries. This means that using a full LU factorization of sparse, multidiagonal matrices with large bandwidths may still end up with large amounts of storage and computations. One group of schemes, known as the Alternating Direction Implicit (ADI) schemes, replaces a multidiagonal matrix by a product of two or more tri-diagonal matrices. More importantly, these schemes maintain the same levels of consistency

u 1 0.5

Figure M.7. Numerical solutions for discontinuous initial condition using the Lax-Wendroff with |η| = 1.

0 1 1 t

0.5 0.5 x 0 0

864

Appendix M: Additional Details and Fortification for Chapter 13

and convergence, as well as the same range of stability as the original schemes. Because the computations are now reduced to solving two or more sequences of tri-diagonal systems, via the Thomas algorithm, the improvements in computational efficiency, in terms of both storage and number of computations, become very significant compared with the direct LU factorizations. The original ADI schemes were developed by Douglas, Peaceman, and Rachford to improve the Crank-Nicholson schemes for parabolic equations. For a simple illustration of the ADI approach, we take the linear second-order diffusion equation for 2D space, without any mixed partial derivatives, given by ∂u ∂t

=

μxx (t, x, y)

∂2u ∂2u ∂u + μ (t, x, y) + βx (t, x, y) yy ∂x2 ∂y2 ∂x

+βy (t, x, y)

∂u + ζ(t, x, y)u + η(t, x, y) ∂y

(M.26)

together with Dirichlet boundary conditions, u(t, 0, y) = v0 (t, y)

;

u(t, x, 0) = w0 (t, y)

u(t, 1, y) = v1 (t, y)

;

u(t, x, 1) = w1 (t, y)

Let u, η, ζ, μxx , μyy , βx , and βy be represented in matrix forms, ⎛ ⎞ ⎞ ⎛ u11 · · · u1N ζ11 · · · ζ1N ⎜ .. ⎟ ; ζ = ⎜ .. .. ⎟ ; etc. .. .. U = ⎝ ... ⎝ . . . . ⎠ . ⎠ uK1

···

uKN

ζK1

···

ζKN

where ukn = u(k x, n y), ζkn = ζ(k x, n y), etc. Following the results of (13.39), the semi-discrete approach yields d v = F(t) v + B(t) dt

(M.27)

where v

=

vec(U)

F

=

Mx

=

My

=

Mx + My  dv  dv 1 μxx IN ⊗ D(2,x) + βx IN ⊗ D(1,x) + ζdv 2 ' (dv ' (dv 1 μyy D(2,y) ⊗ IK + βy D(1,y) ⊗ IK + ζdv 2

B

=

' (dv  dv     T μxx vec B(2,x) + μyy vec B(2,y) +

' (dv  dv       T βx vec B(1,x) + βy − vec [η] vec B(1,y)

and the superscript “dv” is the notation for diagonal-vectorization operation. Applying the Crank-Nicholson scheme, we have     

t (q) (q) t  (q+1)

t (q+1) (q+1) = I+ B (M.28) I− F v + B(q) F v + 2 2 2

Appendix M: Additional Details and Fortification for Chapter 13

865

  By subtracting the term, I − ( t/2) tF(q+1) v(q) , from both sides of (M.28),    t    

t (q+1)  (q+1) I− v F − v(q) = F(q+1) + F(q) v(q) + B(q+1) + B(q) 2 2 (M.29) Let

t v(q) = v(q+1) − v(q) (q+1)

(q+1)

then with F(q+1) = Mx + My   

t (q+1) I−

t v(q) F = 2 =

=

(see (M.28)),   

t (q+1) t (q+1)  I− Mx −

t v(q) My 2 2    

t (q+1)

t (q+1)  I− Mx I−

t v(q) My 2 2   2

t − Mx(q+1) My(q+1) t v(q) 4   (q+1) (q+1) Gx Gy

t v(q) − O t4

where

t (q)

t (q) G(q) ; G(q) Mx (M.30) M x =I − y =I − 2 2 y   The last term is O t4 because of the fact that the Crank-Nicholson schemeguar antees that t v(q) = v(q+1) − v(q) = O t2 . By neglecting terms of order O t4 , (M.29) can then be replaced by   t       F(q+1) + F(q) [u]q + B(q+1) + B(q) Gx(q+1) Gy(q+1) t [u](q) = (M.31) 2 However, Gx and Gy are block tri-diagonal matrices whose nonzero submatrices are diagonal in which the main blocks in the diagonal are also tri-diagonal, thus allowing easy implementation of the Thomas and block-Thomas algorithms. Equation (M.31) is known as the delta-form of the ADI scheme.2 The values of U (q+1) are them obtained from   (M.32) U (q+1) = t U (q) + u(q) It can be shown by direct application of the von Neumann analysis that the ADI scheme given in (M.31) will not change the stability conditions; that is, if the CrankNicholson scheme is unconditionally stable, then the corresponding ADI schemes will also be unconditionally stable. Furthermore, because the only change from the original Crank-Nicholson scheme was the removal of terms that are fourth order in t, the ADI scheme is also consistent. The application of the Lax equivalence theorem then implies that the ADI schemes will be convergent. The extension of the ADI approach to 3D space is straightforward and is given as an exercise. 2

(q)

The scheme is named Alternating Direction Implicit (ADI) based on the fact that the factors Gx (q) and Gy deal separately along the x and y directions, respectively. Also, the term Implicit (the “I” in ADI) is a reminder that ADI schemes are developed to improve the computation of implicit schemes such as the backward-Euler or Crank-Nicholson, where matrix inversions or LU factorizations are required.

866

Appendix M: Additional Details and Fortification for Chapter 13

An important issue with ADI schemes is that for accurate time-marching pro  files, a small time step is still needed. Recall that the removal of the O t4 terms will introduce errors to the original schemes. This additional error is negligible as long as t is chosen small enough. However, time-marching approaches are sometimes used primarily to find steady-state solution. In those cases, accuracy only matters at large time values. Because of stability properties, the errors should then have asymptotically settled out toward zero. The ADI schemes are very often used to obtain steady-state solutions because they handle the complexity and size requirements of 2D and 3D systems efficiently.3

3

Other approaches to steady-state solutions include relaxation methods for solving large sparse linear equations such as Jacobi, Gauss-Seidel, SOR. Currently, various Krylov subspace approaches such as conjugate gradient and GMRES (see Sections 2.7 and 2.8) are used for very large sparse problems.

APPENDIX N

Additional Details and Fortification for Chapter 14

N.1 Convex Hull Algorithm In this section, we describe an algorithm to find a polygonal convex hull of a set of 3D points. The algorithm is a simplified variant of the QuickHull algorithm.1 Furthermore, we restrict the algorithm only to points in three dimensions where all the points are boundary points of the convex hull. This case applies to points that all come from a paraboloid surface. We begin by introducing some terms and operators to be used in the algorithm. 1. Outside sets and visible sets. For a given facet F , let Hyp(F ) be the hyperplane that includes F . Then a point p is outside of F if it is located on the side of Hyp(F ) along with the outward unit normal vector (see Figure N.1). Also, the outside set of F , denoted by Out(F ) = {p 1 , . . . , p  }, is the set of all points that are outside of F . Switching perspectives, for a given point p , a facet F is visible to p if p is outside of F . The visible set of p , denoted by Vis(p ) = {F 1 , . . . , F q }, is the set of all facets that are visible to p . 2. Ridge sets. Let  = {F 1 , . . . , F m } be a set of facets that collectively forms a simply connected region D. Then each boundary edge of D, denoted by Ri , is called a ridge of , and the collection R() = {R1 , R2 , . . . , Rm } is referred to as the ridge set of facets in . For example, from the group of facets shown in Figure N.2, let  = {F 7 , F 8 , F 9 , F 10 , F 13 F 14 F 15 }

1

Barber, C. B., Dobkin, D. B., and Huhdanpaa, H. The QuickHull Algorithm for Convex Hulls. ACM Trans. on Math. Software, 1995.

867

868

Appendix N: Additional Details and Fortification for Chapter 14 F

Hyp(F)

n

a

Figure N.1. Points a and c are outside points of facet F , whereas b and d are not. Hyp(F ) is the hyperplane containing F and n is the outward unit normal vector.

c

b d

then ⎛ ⎜ ⎜ ⎜ ⎜ R() = ⎜ ⎜ ⎜ ⎜ ⎝

(p a , p b) (p b, p c ) (p c , p d ) (p d , p e ) (p e , p f ) (p f , p g ) (p g , p a )

⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠

Note that it will be beneficial for our purposes to specify the sequence of each edge such that they follow a counter-clockwise traversal, for example (p a , p b) instead of (p b, p a ), and so forth. 3. Extend operation. Let p be an outside point to a set of connected facets, . Then the operation Extend(p, ) will take p and attach it to each ridge in R() to form m new facets, where m is the number of ridges of , that is, ⎞ F M+1 ⎟ ⎜ .. ⎠ = Extend(p, ) ⎝ . F M+m ⎛

(N.1)

M is the number of facets before the operation, and F M+i = (p, p i,a , p i,b)

12

4

3

1

9

8

b

10

7

2

13

5

c

g

a

14

d

11 16

15

6

f

e

Figure N.2. The set of facets  = {F 7 , F 8 , F 9 , F 10 , F 13 F 14 F 15 } forms a simply connected region whose edges form the ridge set of .

Appendix N: Additional Details and Fortification for Chapter 14

For example, using the same set  shown in Figure N.2, suppose p h is an outside point to the facets in , then we have ⎛ ⎜ ⎜ ⎜ ⎜ Extend(p,