Last time
=========
* Mutexes -- constrain concurrency (pthread_once does this too)
* Rules
  - Acquire before
  - Hold while !invariant
  - Release after
* Deadlock
  - Circular dependency
  - usually partial order

Btw there are some formal logics for reasoning about these
systems. One: CCS (Concurrent Communication (Communicating?) Systems).

Today
=====
* Condition vars.
  - Constrain order (also pthread_once and pthread_join)
  - Safety in usage
  - Correctness is subtle
* Thread cancellation - want to stop a collection of threads because
  some work is done.
* Per-thread globals (if time)

Aside: hunt down literature about composing locks with counting locks.

Back to our matrix example of waiting for the thread pool, which
looked something like:

mutex_t m;

lock(m);
while(done < count) {
  unlock(m);
  //sleep(0); //omitted today
  lock(m);
}
unlock(m);

Join is probably the right way to accomplish this goal. Now then,
condition variables combine the ability to release a lock to let other
threads do work and also to sleep until something happens. They also
let us communicate information from one thread to another.

Condition variables
Type: pthread_cond_t (abbreviated cond_t or cv_t)
Can initialize statically or dynamically.

Three operations:
* wait(mutex_t l, cond_t c)
  - atomically drops l and sleeps on c.
  - Some time later, thread is "awoken". It reacquires the lock and
    returns. (this is an "exposure point" -- a point where the truth
    of the invariant on the shared state is visible, so the invariant
    had better be true)
* signal(cond_t c)
  - wake up *one* waiting thread (if any)
* broadcast(cond_t c)
  - wake up *all* waiting threads (if any)

The "variable" in condition variable is misleading because they have
no memory. That is, if signal() happens before wait(), the signal() is
lost into the ether. (This is why atomicity of wait() is important.)
Standard doesn't require that you hold a lock when you signal() or
broadcast(), but it's good style. (Failure to do so is a "naked
wakeup", and naked wakeups can sometimes be missed.)

Note that you can't encode the condition you're waiting for and you
can't pass information through the condvar. Here's a simple example.

Imagine the following unsafe third-party datatype (btw, assume any
third-party piece of code is not thread-safe).

class Queue {
public:
    void enqueue(Object & o);
    Object & dequeue();
        // REQUIRES: queue is not empty.
    bool isEmpty();
};

1) Make this thread-safe.
2) Convert dequeue() to a complete function. Dequeue will wait until
   an object is enqueued if necessary.
3) The implementation is not accessible.

[hint hint, exam question]

class SafeQueue {
    Queue myQ;
    pthread_mutex_t myLock;
    pthread_cond_t cond;
public:
    SafeQueue();
    ~SafeQueue(); //discussion deferred
    void enqueue(Object & o);
    Object & dequeue();
};

SafeQueue::SafeQueue()
{
    myLock = PTHREAD_MUTEX_INITIALIZER;

    //didn't read manpage too closely
    cond = PTHREAD_COND_INTIALIZER; 
}

void SafeQueue::enqueue(Object & o)
{
    pthread_lock(myLock);
    //if(myQ.isEmpty()) signal(); //A. this is wrong
    if(myQ.isEmpty()) broadcast(); //B. the correction to A.
    myQ.enqueue(o);
//    signal(cond); //the original inefficient version
    pthread_unlock(myLock);
}

Object & SafeQueue::dequeue()
{
    pthread_lock(myLock);

    while(myQ.isEmpty()) //this cannot be *if* because signal doesn't
    			 //guarantee that the woken thread is next to
    			 //run 
        pthread_wait(myLock, cond);

    Object & r = myQ.dequeue();
    //If we enable option A in enqueue(), can fix it with this instead
    //of B:
    if(!myQ.isEmpty()) signal();
    pthread_unlock(myLock);
}

Condition stealing: we don't know anything about how threads are
schedule. Waking a thread up doesn't mean it's necessarily next to
run. Example:

Thread 1	Thread 2	Thread 3
   .   		   .   		   .
   .   		   .   		   .
   .   		   .   		   .
dequeue wait       .		   .
(blocked)	   .		   .
   .		   enqueue/signal  .
   .		   .		   .
(ready)		   .		   .
   .		   .		dequeue!
   .		   .		   .
try to dequeue     .		   .
   LOSE		   .		   .

There's another issue in the solution (fixed, and highlighted with a
comment). It's an efficiency problem -- signal() is not free. If we do
"if myQ.isEmpty() signal(cond)" before enqueueing, we might have two
threads waiting to dequeue, followed by two enqueues in a row. The
second enqueue will not signal. The solution is to change signal to
broadcast. HOWEVER, is this any more efficient than always signaling?
We'll get spurious wakeups. One spurious wakeup is more expensive than
one spurious signal, so the question of which is more efficient
depends on your expected usage patterns. There's also a way to fix the
original efficiency fix in dequeue().

It ends up being best to just either signal() every time you enqueue
or broadcast() every time you enqueue into an empty queue. Intel TBB
has a class for doing thread-safe real-time counters, and we'll
measure later.

Anywho, what happens when you get read of a SafeQueue with threads
blocked on dequeue? The structure's going away, so the threads also
probably ought to go away as well.

Thread Cancellation

pthread_cancel(tid_t t) - takes thread ID and shoots the thread in the
head. Rules: any thread can cancel any other thread as long as it has
the ID. If you cancel a thread while it's holding a lock, the lock
will always be held. Can't do this arbitrarily -- wait until safe
point.

Btw there's another condition variable operation:

timed_wait(mutex, condvar, timeout).

Remember trylock? It's almost always a bad idea unless there's a
possible circular dependency and there's no good way to enforce a
partial order on the lock. There are even fewer situations when
timed_wait makes sense. You will be tempted to use it when there's a
periodic task that has to happen and you also have to wait to do
something else. The right solution is to use two threads.

<debate about async I/O vs. threads>

Anywho, cancellation. Threads can express whether or not they're
cancellable and, if so, when. 

States:
pthread_cancel_disable -- impervious to cancellation.
pthread_cancel_enable -- can be cancelled, but when?
  - asynchronous (any time)
  - deferred (only when it reaches a cancellation point)

By default, threads are created with the enable deferred state.

Automatic cancellation points:
* pthread_join()
* pthread_cond_wait()
* pthread_cond_timedwait()

(the cancellation points are the functions that are intended to block)

Programmer-defined cancellation points:
pthread_testcancel() - takes no arguments and purpose is to die if
there is a pending cancellation.

To fix the issue at hand (cleanup when threads are blocking on
dequeue)
Add (to SafeQueue)
- private state, list of waiting pthread ids.

In dequeue:
    while(myQ.isEmpty())
    {
         add pthread_self to list
	 wait
	 remove pthread_self from list
    }

Dtor:

lock(myLock)
for each waiting thread
    cancel the thread
unlock(myLock)