Last Time:

(XXX: I (notetaker) was late and got here halfway through the
summmary)

Today:

* Continue discussion of cancellation: cancellation stacks (for
  cleanup on cancellation)
* Scheduling priority
* Priority inversion (a problem)

Assignment: build thread-safe queue, build application of your
choosing using the queue stuff, explain how you're testing. (will be
handed out officially)

Envisioned timeline: fuzzy idea of project before break, firm proposal
shortly after break, then juuuust work on project. We'll do a threads
project later this week, and then a TBB project later this week.

Anywho, what if we cancel a thread but it has other dynamic resources
that need to be cleaned up? We have the cancel stack! It's a stack of
cleanup routines that threads can push onto, and the routines on the
stack are called when the thread is cancelled or exits.

Type of cleanup routine: void (*fn)(void *)

pthread_cleanup_push(fn, arg)
pthread_cleanup_pop(fn, arg)

These must happen in pairs in the same lexical scope.

Could combine cleanup with exceptions by having your cleanup routine
throw an exception (to avoid duplicating the cleanup code in the
cleanup routine AND the exception handler).

Example cleanup code:

thread ()
{
char * bf = new char [2555];
pthread_cleanup_push(delBuf, buf);

...

delete [] buf;
pthread_cleanup_pop(0);
}

problem: if you're cancelled in the middle of the cleanup code, you're
screwed. Therefore, you need to turn off cancellation temporarily to
get this stuff working.




Anywho, priorities. How your operating system schedules tasks today:
It uses Shortest Time to Completion First. (Completely Fair Scheduler
in Linux being totally off-topic, apparently) To implement this, we
just guess at time to completion with priorities. The operating system
uses a set of priority queues. It runs the highest priority ready
task, and if it finishes before its timeslice is up then it gets a
higher prioirty. If the thread uses its whole timeslice, it's demoted
in priority. Usually, threads use the system scheduler directly, but
some implementations have extras:

1) Set priority level directly
2) Disable promotion/demotion mechanism
3) Disable preemption (!)

all on a per-thread basis. Threads with real-time deadlines have high
priority, and housekeeping stuff (garbage collection?) tends to be
low. We don't bother with this unless we really need to tune
performance.

By the way, the number of discrete priorities is
implementation-dependent, so you should probably use the pthreads
functions that allow you to obtain the max and min priorities.

Scheduling disciplines:
OTHER - whatever the kernel would normally do. Typically don't
manipulate priority.

ROUND_ROBIN - preemptible but fixed priority.

FIFO - non-preemptible and fixed priority.

(From the manpage for pthread_getschedparam(3): Thread has to have
root to schedule as RR or FIFO because they're "realtime".)

Even nonpreemtive threads can be stopped by higher priority threads.

Short, periodic real-time tasks are typically given high priority and
FIFO scheduling.

Using priorities well is an art. It's probably more common to assign
lower than default priorities to threads. For example, maybe I have a
big tree structure in memory, and a low-priority thread is assigned to
rebalance the tree. Sadly, we have to protect the tree with a mutex to
do this. Suppose that the low-priority thread starts
rebalancing. Sometime later, a high-priority thread wakes up and tries
to acquire the mutex, but it can't. Then, a medium-priority thread
having nothing to do with the tree wakes up and gets to run. Now, THE
MEDIUM-PRIORITY THREAD IS HOLDING UP THE HIGH-PRIORITY THREAD!
(indirectly, by holding up the low-priority thread) This is priority
inversion.

Pthreads has 2 mechanisms to deal with priority inversion. The first
is the *priority ceiling*: we associate a priority with a mutex. Any
thread holding a mutex has priority max(its normal priority, the mutex
priority). (This should probably be called a priority floor?) The
problem is that this might give too much -- the thread gets boosted in
priority whether or not a more important thread is waiting on the
mutex.

The second technique is *priority inheritance*: a thread holding a
mutex gets at least the priority of the highest priority thread
blocked on that mutex.

This is all very cute, but we wait for things that aren't mutexes:
signals, joins, etc. Bummer.

Extra bonus feature of pthreads: sometimes when you're converting
nonthreaded code to threaded code, there are globals that you don't
really want to share between threads. Can use the key: bind "names" to
per-thread state. This is almost always a bad idea if you have a
choice (i.e., you're starting from scratch).

Hooray, we're probably done talking about pthreads mechanisms. We'll
be transitioning to "structuring things in a task-parallel way", which
means we should be referring to the parallel patterns book.