Parallel Programming FS23

Time & Location

Contact

  • Name: Andreas Ellison
  • Email: aellison@student.ethz.ch
  • Discord: HumbleCatcher#1527
    (I prefer this over email, don't hesitate to ask me questions πŸ™‚)

Resources

Part 1

Part 2

  • Overall, "The Art of Multiprocessor Programming" is a very very good resource for this part of the course. It is very well written and covers almost all topics of the lecture in depth.
    • Chapter 2 (p. 21): Covers the mutual exclusion algorithms from the lecture, with proofs.
    • Chapter 7, 7.1-7.4 (p. 141): Covers TAS, TATAS and exponential backoff locks.
    • Chapter 8 (p. 177): Covers monitors, semaphores and the readers-writers lock, but a bit differently to the lecture.
    • Chapter 9 (p. 195): The content on lock granularity from the lecture is based on this chapter in the book and follows it very closely. The book is a nice reference here especially for implementations of the algorithms.
    • Chapter 10, 10.5-10.6 (p. 230): Covers the implementation of a lock-free unbounded queue and the ABA problem, using the queue as an example.
    • Chapter 3, 3.1-3.6 (p. 45): Covers sequential consistency and linearizability (and quiescent consistency, but this is not exam relevant). The proofs starting from 3.6.2 were not covered in the lecture (and are hence of course not exam relevant). Good to know: the book always lists the linearization points of code given later chapters.
    • Chapter 5, excluding 5.1.1 and 5.5-7 (p. 99): Covers consensus. Proofs like for theorem 5.2.1 or 5.4.1 are not exam relevant.
    • Chapter 18 (p. 417): Covers transactional memory. Parts of the chapter go a bit more in depth into implementation details that were not covered in the lecture.
  • Resources on the Java memory model:
    • Short summary of happens-before from the Java documentation.
    • Official paper explaining the memory model. It should be understandable. Alternatively, you can read the official specification, which is more compact but has less explanations. If you choose the specification, I can recommend reading up to and including section 17.4.5.
    • A nice talk on the Java memory model.
    • Chapter 16.1 (p. 337) of the book "Java Concurrency in Practice" has an explanation of the Java memory model.
    • An introductory blog article on memory models in general.
  • Resources on MPI:

FAQ

Exercise sessions

Week 2

  • "Derivation" for why we need a scheduler, processes and threads
  • Part two, chapters 3 and 4 of this book are about processes and threads, if you want to read more about the topic.
    Note: Having a deep understanding of processes and threads in an OS is not exam relevant for this course.
  • Demo (Eclipse project) of summing up the values in an array with Java threads
  • Quiz

Week 3

Week 4

  • My notes on parallelism vs. concurrency and pipelining
  • HS20 wait and notify task: code template and link to exam
  • Plots shown in class and the code used to generate them (first run the Java program, then run the script plot.py). Please let me know if you find any mistakes in the code to generate the data!

The first few people to email me a solution to the following challenge before

  • 22.03.2023 4 PM for the Wednesday group
  • 24.03.2023 10 AM for the Friday group

get a bar of chocolate. 🍫

Prove that for an arbitrary pipeline with exactly one "execution unit" per stage (e. g. if one stage were to use the washing machine, then there would only be one washing machine available) the following holds:

the pipeline is balanced

β€…β€ŠβŸΊβ€…β€Š\iff
no stage is longer than the first

Note: A pipeline is balanced if all iterations have the same latency.
Solution: See notes for week 5.

Week 5

  • My proofs on pipelining
  • The chapters in the PVW script on Amdahl's and Gustafson's laws might be helpful for the exercises

Week 6

  • My notes on Amdahl's and Gustafson's laws. The explanation in the exercise session slides (on Moodle) is also good.
  • Kahoot and short explanations for some of the pipelining questions.

The following resources are interesting if you're curious about the history behind Amdahl's and Gustafson's "laws" and why they exist.

  • The paper that introduced Gustafson's law (relatively easy to read)
  • History behind Amdahl's law (you will need to use your ETH login to access the article)

Week 7

Please fill out the feedback form!

Some advanced resources, beyond the scope of the course:

  • Paper on how the Fork/Join framework is implemented in Java on a high level
  • Chapter 16 of this book is a much more advanced resource on task parallelism, if you want to have a peek at what more there is to the topic (e.g. it discusses the most relevant theoretical results and gives an implementation of a work-stealing scheduler in Java).
  • A theoretical paper on an algorithm for work-stealing scheduling which was referenced in the lecture slides

Week 8

  • The livelock example from class.
  • Demo code of a CAS lock and the plots I showed in class. The
  • The Java program with unexpected behavior that I showed at the end.
  • There are links to some useful resources on the current topics at the top of the page (under "Part 2").

Week 9

  • There are links to some useful resources on the current topics at the top of the page (under "Part 2").
  • Proofs showing that the Filter lock satisfies mutual exclusion and starvation freedom can be found on page 30 of "The Art of Multiprocessor Programming". The same can be found for the bakery lock later on.

Week 10

Types of exercises that might come up in the exam
Disclaimer: This list is not guaranteed to be complete and is only meant to give you an idea of what has been asked on previous exams.
Locks
Usually there are not too many question on this topic.
  • true/false questions of which lock has which properties (fairness, starvation free)
  • find bug in lock code (violation of mutual exclusion or deadlock freedom)
  • draw state space diagram and/or read off correctness properties
  • reproduce Peterson/Filter/Bakery lock
  • prove correctness of Peterson lock or similar (but not Filter or Bakery)
Monitors, semaphores, barriers
  • semaphore implementation (mostly with monitors)
  • (never seen rendezvous with semaphores in an exam)
  • barrier implementation (mostly with monitors)
  • (only seen a task on implementing a barrier with semaphores once in FS21, 8b)
  • fill out some program using monitors (similar to wait/notify exercises, maybe with lock conditions)
What we did in class
  • Barriers:
    • Code for a non-reusable barrier using wait/notify.
    • Exercises on correctness of barrier implementations: FS19, 11a, HS14 3c (there was no time for this during the session)
    • Code for first (more recommended for exam) and second (more elegant, but less convenient for the exam because of the use of ThreadLocal) implementation of a reentrant barrier using wait/nofity.
  • Kahoot

Week 11

  • Kahoot
  • Up to date solution for why the Peterson lock works in Java.
  • I can very much recommend the book chapter on "lock granularity" from "The Art of Multiprocessor Programming" (see resources above) for further reference on the topic. It has code for most of the operations, so it's especially useful if you're stuck with this week's assignment.

Week 12

  • The explanation for race condition vs. bad interleaving vs. data race can be found in the new "FAQ" section above.
  • Quiz with slides. Credit to HansjΓΆrg Schraubenzieher for the questions.
  • Little derivation of enq from the lock-free unbounded queue implementation of the lecture. (I didn't show this on Wednesday.)

Week 13

  • For an example of how the ABA problem can happen with the lock-free unbounded queue, see chapter 10.6 of the book
  • My notes on sequential consistency and linearizability. I am more than happy to answer questions about my explanations, e.g. if it is unclear how everything connects to the way it is presented in the lecture/book. Also the book covers the topic way back in chapter 3 (3.1 - 3.6) and if you want to read deeper into the topic, this is the paper that initially introduced the concept. (You may recognize one of the authors.)

Week 14

  • I added an explanation of linearization points to the "FAQ" section above.
  • Check the resources at the top for where to read more about consensus and transactional memory in the book and for some resources on MPI.
  • Quiz (with explanations for the consensus questions)
Exam tips for consensus and MPI
Disclaimer: The following tips are only my thoughts as a student. The lecture makes no such promises and according to them everything that was covered in the lecture is exam relevant (unless stated otherwise).
  • Consensus:
    • I have a strong feeling that the head TA wants to ask this in the exam. ;)
    • I can think of three types of questions on this topic: short questions on the definitions, identifying whether given code is a correct consensus protocol or making a statement about the consensus number of an object. For the latter, I only see two options:
      1. Proving that an object x has consensus number
        β‰₯y\ge y
        (or
        ∞\infty
        ). Then you must provide an algorithm solving
        yy
        -thread consensus using any number of instances of x (or for the
        ∞\infty
        case, provide an algorithm solving
        nn
        -thread consensus for arbitrary
        nn
        ).
      2. Proving that an object x has consensus number at most
        zz
        . This would involve proving that it is impossible to implement
        (z+1)(z + 1)
        -thread consensus with x. With the content of the lecture, the only option I see is to use the theorems that were shown. For example, if we know that x was implemented using atomic registers, then it cannot solve consensus for more than one thread. Proofs that do not rely on such results require some concepts that were not covered in the lecture. (An example is the proof for why atomic registers cannot solve 2-thread consensus in "The Art of Multiprocessor Programming" (p. 103).)
      So I would recommend to practice writing consensus protocols and revising the results from the lecture.
  • MPI: Since the topic was only covered in short time in the lecture and no exercises were provided on the topic, I would not expect coding exercises with MPI in the exam (again, cannot promise). But that does not exclude reading and understanding MPI code. I would make sure to remember what the collective operations do, as this has been asked before.