随机向量(Random Vectors)
考虑取n n n X i : Ω → R X_i:\Omega\to\R X i : Ω → R X = ( X 1 , ⋯ , X n ) X=(X_1,\cdots,X_n) X = ( X 1 , ⋯ , X n ) E [ X ] = ( E [ X 1 ] , ⋯ , E [ X n ] ) \mathbb{E}[X]=(\mathbb{E}[X_1],\cdots,\mathbb{E}[X_n]) E [ X ] = ( E [ X 1 ] , ⋯ , E [ X n ])
协方差(Covariance)
对于随机变量，我们定义过方差Var ( X ) = E [ ( X − E [ X ] ) 2 ] \text{Var}(X)=\mathbb{E}[(X-\mathbb{E}[X])^2] Var ( X ) = E [( X − E [ X ] ) 2 ] E [ ( X − E [ X ] ) ( X − E [ X ] ) ⊤ ] \mathbb{E}[(X-\mathbb{E}[X])(X-\mathbb{E}[X])^\top] E [( X − E [ X ]) ( X − E [ X ] ) ⊤ ] n × n n\times n n × n Cov ( X ) \text{Cov}(X) Cov ( X )
根据定义，我们可以计算协方差矩阵第i i i j j j Cov ( X ) i j = E [ ( X i − E [ X i ] ) ( X j − E [ X j ] ) ] \text{Cov}(X)_{ij}=\mathbb{E}[(X_i-\mathbb{E}[X_i])(X_j-\mathbb{E}[X_j])] Cov ( X ) ij = E [( X i − E [ X i ]) ( X j − E [ X j ])] Cov ( X i , Y i ) \text{Cov}(X_i,Y_i) Cov ( X i , Y i ) Var ( X ) = E [ ( X − E [ X ] ) 2 ] = Cov ( X , X ) \text{Var}(X)=\mathbb{E}[(X-\mathbb{E}[X])^2]=\text{Cov}(X,X) Var ( X ) = E [( X − E [ X ] ) 2 ] = Cov ( X , X )
对于两个变量X , Y X,Y X , Y Cov ( X , Y ) = E [ ( X − E [ X ] ) ( Y − E [ Y ] ) ] \text{Cov}(X,Y)=\mathbb{E}[(X-\mathbb{E}[X])(Y-\mathbb{E}[Y])] Cov ( X , Y ) = E [( X − E [ X ]) ( Y − E [ Y ])] = E [ X Y ] − E [ X E [ Y ] ] − E [ E [ X ] Y ] + E [ E [ X ] E [ Y ] ] =\mathbb{E}[XY]-\mathbb{E}[X\mathbb{E}[Y]]-\mathbb{E}[\mathbb{E}[X]Y]+\mathbb{E}[\mathbb{E}[X]\mathbb{E}[Y]] = E [ X Y ] − E [ X E [ Y ]] − E [ E [ X ] Y ] + E [ E [ X ] E [ Y ]] = E [ X ] E [ Y ] − E [ X ] E [ Y ] − E [ X ] E [ Y ] + E [ X ] E [ Y ] = 0 =\mathbb{E}[X]\mathbb{E}[Y]-\mathbb{E}[X]\mathbb{E}[Y]-\mathbb{E}[X]\mathbb{E}[Y]+\mathbb{E}[X]\mathbb{E}[Y]=0 = E [ X ] E [ Y ] − E [ X ] E [ Y ] − E [ X ] E [ Y ] + E [ X ] E [ Y ] = 0 X = ( X 1 , ⋯ , X n ) X=(X_1,\cdots,X_n) X = ( X 1 , ⋯ , X n ) n n n Cov ( X i , X j ) = 0 , Cov ( X i , X i ) = Var ( X i ) \text{Cov}(X_i,X_j)=0,\text{Cov}(X_i,X_i)=\text{Var}(X_i) Cov ( X i , X j ) = 0 , Cov ( X i , X i ) = Var ( X i ) Cov ( X ) \text{Cov}(X) Cov ( X ) i i i i i i X i X_i X i
对于任意随机向量X X X Cov ( X ) i j = Cov ( X i , X j ) = E [ ( X i − E [ X i ] ) ( X j − E [ X j ] ) ] \text{Cov}(X)_{ij}=\text{Cov}(X_i,X_j)=\mathbb{E}[(X_i-\mathbb{E}[X_i])(X_j-\mathbb{E}[X_j])] Cov ( X ) ij = Cov ( X i , X j ) = E [( X i − E [ X i ]) ( X j − E [ X j ])] = E [ ( X j − E [ X j ] ) ( X i − E [ X i ] ) ] = Cov ( X j , X i ) = Cov ( X ) j i =\mathbb{E}[(X_j-\mathbb{E}[X_j])(X_i-\mathbb{E}[X_i])]=\text{Cov}(X_j,X_i)=\text{Cov}(X)_{ji} = E [( X j − E [ X j ]) ( X i − E [ X i ])] = Cov ( X j , X i ) = Cov ( X ) j i
对于任意随机向量X X X Cov ( X ) \text{Cov}(X) Cov ( X ) ∀ x ∈ R n \forall x\in \R^n ∀ x ∈ R n x ⊤ Cov ( X ) x = x ⊤ E [ ( X − E [ X ] ) ( X − E [ X ] ) ⊤ ] x x^\top \text{Cov}(X)x=x^\top \mathbb{E}[(X-\mathbb{E}[X])(X-\mathbb{E}[X])^\top]x x ⊤ Cov ( X ) x = x ⊤ E [( X − E [ X ]) ( X − E [ X ] ) ⊤ ] x x x x = E [ x ] ⊤ E [ ( X − E [ X ] ) ( X − E [ X ] ) ⊤ ] E [ x ] = E [ x ⊤ ( X − E [ X ] ) ( X − E [ X ] ) ⊤ x ] =\mathbb{E}[x]^\top\mathbb{E}[(X-\mathbb{E}[X])(X-\mathbb{E}[X])^\top]\mathbb{E}[x]=\mathbb{E}[x^\top (X-\mathbb{E}[X])(X-\mathbb{E}[X])^\top x] = E [ x ] ⊤ E [( X − E [ X ]) ( X − E [ X ] ) ⊤ ] E [ x ] = E [ x ⊤ ( X − E [ X ]) ( X − E [ X ] ) ⊤ x ] = E [ ( ( X − E [ X ] ) ⊤ x ) 2 ] ≥ 0 =\mathbb{E}[((X-\mathbb{E}[X])^\top x)^2]\geq 0 = E [(( X − E [ X ] ) ⊤ x ) 2 ] ≥ 0 ∀ x , x ⊤ Cov ( X ) x ≥ 0 \forall x,x^\top \text{Cov}(X)x\geq 0 ∀ x , x ⊤ Cov ( X ) x ≥ 0 Cov ( X ) \text{Cov}(X) Cov ( X )
多元正态分布(Multivariate Normal Distribution)
称随机变量满足正态分布(normal distribution)（或高斯分布(Gaussion distribution），记为X ∼ N ( μ , σ 2 ) X \sim N(\mu,\sigma^2) X ∼ N ( μ , σ 2 ) X X X f ( x ) = 1 2 π σ 2 e − ( x − μ ) 2 2 σ 2 f(x)=\dfrac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(x-\mu)^2}{2\sigma^2}} f ( x ) = 2 π σ 2 1 e − 2 σ 2 ( x − μ ) 2 X ∼ N ( μ , σ 2 ) X\sim N(\mu,\sigma^2) X ∼ N ( μ , σ 2 ) E [ X ] = μ , Var [ X ] = σ 2 \mathbb{E}[X]=\mu,\text{Var}[X]=\sigma^2 E [ X ] = μ , Var [ X ] = σ 2 μ \mu μ σ 2 \sigma^2 σ 2 N ( μ , σ 2 ) N(\mu,\sigma^2) N ( μ , σ 2 )
对于n n n X X X n × n n\times n n × n A A A n n n μ \mu μ n n n N ( 0 , 1 ) N(0,1) N ( 0 , 1 ) ξ = ( ξ 1 , ⋯ , ξ n ) \xi=(\xi_1,\cdots,\xi_n) ξ = ( ξ 1 , ⋯ , ξ n ) X = A ξ + μ X=A\xi+\mu X = A ξ + μ X X X
对于多元正态分布X = A ξ + μ X=A\xi+\mu X = A ξ + μ E [ X ] = E [ A ξ ] + E [ μ ] = μ \mathbb{E}[X]=\mathbb{E}[A\xi]+\mathbb{E}[\mu]=\mu E [ X ] = E [ A ξ ] + E [ μ ] = μ Cov ( X ) = E [ ( X − E [ X ] ) ( X − E [ X ] ) ⊤ ] \text{Cov}(X)=\mathbb{E}[(X-\mathbb{E}[X])(X-\mathbb{E}[X])^\top] Cov ( X ) = E [( X − E [ X ]) ( X − E [ X ] ) ⊤ ] = E [ ( A ξ ) ( A ξ ) ⊤ ] =\mathbb{E}[(A\xi)(A\xi)^\top] = E [( A ξ ) ( A ξ ) ⊤ ] = E [ A ξ ξ ⊤ A ⊤ ] =\mathbb{E}[A\xi\xi^\top A^\top] = E [ A ξ ξ ⊤ A ⊤ ] A A A Cov ( X ) = A E [ ξ ξ ⊤ ] A ⊤ = A E [ ( ξ − E [ ξ ] ) ( ξ − E [ ξ ] ) ⊤ ] A ⊤ = A Cov ( ξ ) A ⊤ \text{Cov}(X)=A\mathbb{E}[\xi\xi^\top ]A^\top=A\mathbb{E}[(\xi-\mathbb{E}[\xi])(\xi-\mathbb{E}[\xi])^\top]A^\top=A\text{Cov}(\xi)A^\top Cov ( X ) = A E [ ξ ξ ⊤ ] A ⊤ = A E [( ξ − E [ ξ ]) ( ξ − E [ ξ ] ) ⊤ ] A ⊤ = A Cov ( ξ ) A ⊤ = A I n A ⊤ = A A ⊤ =AI_nA^\top=AA^\top = A I n A ⊤ = A A ⊤ Cov ( X ) = A A ⊤ \text{Cov}(X)=AA^\top Cov ( X ) = A A ⊤ Σ \Sigma Σ X = A ξ + μ X=A\xi+\mu X = A ξ + μ X ∼ N ( μ , Σ ) X\sim N(\mu,\Sigma) X ∼ N ( μ , Σ )
可以用多元积分计算得到多元正态分布的density f ( x ) = 1 ( 2 π ) n / 2 ∣ det Σ ∣ 1 / 2 exp ( − 1 2 ( x − μ ) ⊤ Σ − 1 ( x − μ ) ) f(x) = \dfrac{1}{(2\pi)^{n/2} |\det \Sigma|^{1/2}} \exp\left(-\dfrac{1}{2} (x - \mu)^\top \Sigma^{-1} (x - \mu)\right) f ( x ) = ( 2 π ) n /2 ∣ det Σ ∣ 1/2 1 exp ( − 2 1 ( x − μ ) ⊤ Σ − 1 ( x − μ ) )