Merge pull request #5 from Arman-Beykmohammadi/w12_solutions

Assignment 1 and 2 solutions added to the book

Author: clippert

book/_toc.yml

@@ -13,7 +13,7 @@ parts:
     chapters:
     # week 1
       - file: chapter_ml_basics/intro
-        title: Machine Learning Basics
+        title: Machine Learning Problems
         sections:
         - file: chapter_ml_basics/classification
           title: Classification
@@ -66,25 +66,45 @@ parts:
         - file: chapter_calculus/minima_first_order_condition
           title: First Order Condition
         - file: chapter_calculus/analytical_solution_ridge
-          title: Ridge Regression
+          title: Quadratic Optimization
         - file: chapter_calculus/line_search
           title: Line Search
         - file: chapter_calculus/hessian
           title: Hessian
+        - file: chapter_calculus/taylors_theorem
+#        - file: chapter_calculus/irls
+#          title: Iteratively Re-Weighted Least Squares
 # study the properties of matrices
-#  - file: chapter_decompositions/overview_decompositions # chapter_linear_algebra/linear_algebra
-#    sections:
-#    - file: chapter_decompositions/eigenvectors
-#    - file: chapter_decompositions/trace_determinant
-#    - file: chapter_decompositions/orthogonal_matrices
-#    - file: chapter_decompositions/symmetric_matrices
-#    - file: chapter_decompositions/psd_matrices
-#    - file: chapter_decompositions/svd
-#    - file: chapter_decompositions/big_picture
-#    - file: chapter_decompositions/pseudoinverse
-#    - file: chapter_decompositions/low_rank_approximation
-#    - file: chapter_decompositions/matrix_norms
+      - file: chapter_decompositions/overview_decompositions
+        title: Matrix Analysis   
+        sections:
+        - file: chapter_decompositions/matrix_rank
+        - file: chapter_decompositions/determinant
+        - file: chapter_decompositions/row_equivalence
+        - file: chapter_decompositions/square_matrices
+        - file: chapter_decompositions/trace
+        - file: chapter_decompositions/eigenvectors # end week 05
+        - file: chapter_decompositions/orthogonal_matrices
+        - file: chapter_decompositions/symmetric_matrices
+        - file: chapter_decompositions/Rayleigh_quotients 
+        - file: chapter_decompositions/matrix_norms
+        - file: chapter_decompositions/psd_matrices
+        - file: chapter_decompositions/pca  # PCA as example for the eigenvalue decomposition of a psd matrix
+          title: Principal Components Analysis
+        - file: chapter_decompositions/svd # 
+#        - file: chapter_decompositions/RBF_kernel_Positive_Definite
+        - file: chapter_decompositions/pseudoinverse
+        - file: chapter_decompositions/orthogonal_projections
+        - file: chapter_decompositions/big_picture
+          title: Fundamental Subspaces
+#        - file: chapter_decompositions/representer_theorem
+#      - file: chapter_convexity/overview_convexity
+#        title: Convexity
+#        sections:
+#        - file: chapter_convexity/convex_sets
+#        - file: chapter_convexity/convex_functions
 # continue with second order optimization
+#      title: Second-Order Optimization
 #        - file: chapter_calculus/newtons_method
 #          title: Newton's Method
 #        - file: chapter_taylor/minima_second_order_condition
@@ -93,16 +113,13 @@ parts:
 #        - file: chapter_calculus/orthogonal_projections
 #      - file: chapter_taylor/overview_taylor
 #        sections:
-#  - file: chapter_convexity/overview_convexity
-#    sections:
-#    - file: chapter_convexity/convexity
 #  - file: chapter_optimization/overview_optimization
 #    sections:
 #    - file: chapter_optimization/optimization
 #    - file: chapter_optimization/optimization_second_order
 #    - file: chapter_optimization/bfgs
-#    - file: chapter_optimization/orthogonal_projection
 #  - file: chapter_probability/overview_probability
+#    title: Probability and Random Variables
 #    sections:
 #    - file: chapter_probability/probability_basics
 #    - file: chapter_probability/random_variables
@@ -149,8 +166,21 @@ parts:
             title: First Fundamental Theorem of Calculus
           - file: appendix/second_fundamental_theorem_calculus
             title: Second Fundamental Theorem of Calculus
+          - file: appendix/Clairauts_theorem
+            title: Clairaut's Theorem
       - file: appendix/differentiation_rules
         title: Differentiation Rules
+      - file: appendix/Exercise Sheet Solutions.md
+        title: Exercise Sheet Solutions
+        sections:
+          - file: appendix/Exercise Sheet 1 Solutions.md
+            title: Exercise Sheet 1 Solutions
+          - file: appendix/Exercise Sheet 2 Solutions.md
+            title: Exercise Sheet 2 Solutions
+          - file: appendix/Exercise Sheet 3 Solutions.md
+            title: Exercise Sheet 3 Solutions
+          - file: appendix/Exercise Sheet 4 Solutions.md
+            title: Exercise Sheet 4 Solutions
 #        sections:
 #          - file: appendix/proof_vector_spaces
 #            title: Vector Spaces

book/appendix/Clairauts_theorem.md

@@ -0,0 +1,77 @@
+# Symmetry of Mixed Partial Derivatives (Clairaut’s Theorem)
+
+:::{prf:theorem} Clairaut Schwarz
+:label: thm-Clairaut-appendix
+:nonumber:
+
+Let $f: \mathbb{R}^2 \to \mathbb{R}$ be a function such that both mixed partial derivatives $\frac{\partial^2 f}{\partial x \partial y}$ and $\frac{\partial^2 f}{\partial y \partial x}$ exist and are **continuous** on an open set containing a point $(x_0, y_0)$
+
+Then:
+
+$$
+\boxed{
+\frac{\partial^2 f}{\partial x \partial y}(x_0, y_0) = \frac{\partial^2 f}{\partial y \partial x}(x_0, y_0)
+}
+$$
+
+That is, **the order of differentiation can be interchanged**.
+:::
+
+## Intuition
+
+If a function is smooth enough (specifically, if the second-order partial derivatives exist and are continuous), then the "curvature" in the $x$ direction after differentiating in the $y$ direction is the same as the curvature in the $y$ direction after differentiating in the $x$ direction.
+
+---
+
+## Proof Sketch
+
+We will sketch a proof using the **mean value theorem** and the definition of partial derivatives. Let’s assume that $f$ has continuous second partial derivatives in an open rectangle around the point $(x_0, y_0)$.
+
+Define:
+
+$$
+F(h,k) = \frac{f(x_0 + h, y_0 + k) - f(x_0 + h, y_0) - f(x_0, y_0 + k) + f(x_0, y_0)}{hk}
+$$
+
+Then, as $h, k \to 0$, $F(h,k) \to \frac{\partial^2 f}{\partial y \partial x}(x_0, y_0)$ and also $F(h,k) \to \frac{\partial^2 f}{\partial x \partial y}(x_0, y_0)$, provided the second partial derivatives are continuous.
+
+### Step-by-step:
+
+1. By the **Mean Value Theorem**, the numerator of $F(h,k)$ can be interpreted as a finite difference approximation to a mixed partial derivative.
+2. Using Taylor’s Theorem with remainder, or via integral representations of derivatives, one can show that:
+
+   $$
+   \lim_{(h,k) \to (0,0)} F(h,k) = \frac{\partial^2 f}{\partial x \partial y}(x_0, y_0)
+   $$
+
+   and also
+
+   $$
+   \lim_{(h,k) \to (0,0)} F(h,k) = \frac{\partial^2 f}{\partial y \partial x}(x_0, y_0)
+   $$
+
+   due to continuity of the second derivatives.
+3. Hence, the limits agree and the mixed partials are equal.
+
+Therefore:
+
+$$
+\frac{\partial^2 f}{\partial x \partial y}(x_0, y_0) = \frac{\partial^2 f}{\partial y \partial x}(x_0, y_0)
+$$
+
+---
+
+## When Clairaut's Theorem **Does Not Apply**
+
+If the second-order mixed partial derivatives exist but are **not continuous**, the symmetry may fail. A classic counterexample is:
+
+$$
+f(x, y) =
+\begin{cases}
+\frac{xy(x^2 - y^2)}{x^2 + y^2}, & \text{if } (x, y) \neq (0, 0) \\
+0, & \text{if } (x, y) = (0, 0)
+\end{cases}
+$$
+
+This function has both mixed partial derivatives at the origin, but they are not equal because they are not continuous there.
+

book/appendix/Exercise Sheet 1 Solutions.md

@@ -0,0 +1,60 @@
+# Exercise Sheet 1 Solutions
+
+
+### 1.
+#### (a)
+Take any \(v_1=(a,b)\) and \(v_2=(c,d)\) in \(V\); then \(b=3a+1\) and \(d=3c+1\).  
+Their sum is  
+\[
+v_1+v_2=(a+c,\;b+d)=(a+c,\;3a+1+3c+1)=\bigl(a+c,\;3(a+c)+2\bigr),
+\]  
+which **does not** satisfy \(b+d=3(a+c)+1\). Hence \(V\) is *not* closed under addition ⇒ **not a vector space**.  
+(Equivalently, the additive identity \((0,0)\notin V\), violating axiom V1.)
+
+#### (b)
+All axioms except **distributivity over scalar addition** fail:
+
+Take \(v=(a,b)\) and scalars \(\alpha,\beta\in\mathbb R\).
+\[
+(\alpha+\beta)\,v=((\alpha+\beta)a,\;b),
+\quad
+\alpha v+\beta v=(\alpha a,\;b)+(\beta a,\;b)=((\alpha+\beta)a,\;2b).
+\]
+Unless \(b=0\), the second component differs, so  
+\((\alpha+\beta)v\neq\alpha v+\beta v\).  
+Therefore \(V\) is **not** a vector space.
+
+
+### 2.
+#### (a)
+*Zero vector:* \((0,0)\) satisfies \(0=2\cdot0\).  
+*Closure (addition):* if \(y_1=2x_1\) and \(y_2=2x_2\), then
+\[
+y_1+y_2 = 2(x_1+x_2).
+\]
+*Closure (scalar mult.):* for \(\alpha\in\mathbb R\),
+\[
+\alpha(x,y)=(\alpha x,\;2\alpha x).
+\]
+All three conditions hold ⇒ \(W\) **is a subspace**.
+
+#### (b)
+Pick \((x,y)\in W\) with \(x>0\) and any negative scalar \(\alpha<0\).  
+Then
+\[
+\alpha(x,y)=(\alpha x,\;\alpha y),
+\]
+and \(\alpha x<0\). Thus \(\alpha(x,y)\notin W\).  
+Not closed under scalar multiplication ⇒ **not a subspace**.
+
+
+### 3.
+For \(x=(a,b)\), \(y=(c,d)\) and scalars \(\alpha,\beta\):
+\[
+T(\alpha x+\beta y)=\bigl((\alpha a+\beta c)^{2},\;\alpha b+\beta d\bigr),
+\]
+\[
+\alpha T(x)+\beta T(y)=\bigl(\alpha^{2}a^{2}+\beta^{2}c^{2},\;\alpha b+\beta d\bigr).
+\]
+The first components differ unless \(a c=0\) or \(\alpha\beta=0\).  
+Hence \(T\) **violates additivity/homogeneity ⇒ not linear**.