The class StatisticsNumberSet looks very good, the only thing I would change is that ugly switch-case in getStandardDeviation, you could replace it by a single and more readable if:

   if (numberOfElements < 2) {
    return Double.NAN;
    }

Now in my opinion you should improve your test suit.
First of all there sould be one assertion per test, you should test only one logical thing. In this way, when a test fail it will be easier for you to locate the code that caused the failure. Try to keep your tests as small and fast as possible:

@Test
public void testEmptySet() {
    assertEquals(0, set.size());  
}

@Test
public void testAdd_double() {
    int elementsBefore = set.size();
    set.add(1.0);
    assertEquals(elementsBefore + 1, set.size());
}

@Test
public void testRemove_Number() {
    set.add(10.0);
    int sizeBefore = set.size();
    set.remove(10.0);
    assertEquals(sizeBefore - 1, set.size());
}

You could write more tests, for example you could test your exceptions:

@Test(expected = IllegalArgumentException.class)
public void testAddingNaN() {
    set.add(Double.NaN);
}

Try to use a code coverage tool to see how much code your test cover.

The class StatisticsNumberSet looks very good, the only thing I would change is that ugly switch-case in getStandardDeviation, you could replace it by a single and more readable if:

   if (numberOfElements < 2) {
    return Double.NAN;
    }

Now in my opinion you should improve your test suit.
First of all there sould be one assertion per test, you should test only one logical thing. In this way, when a test fail it will be easier for you to locate the code that caused the failure. Try to keep your tests as small and fast as possible:

@Test
public void testEmptySet() {
    assertEquals(0, set.size());  
}

@Test
public void testAdd_double() {
    int elementsBefore = set.size();
    set.add(1.0);
    assertEquals(elementsBefore + 1, set.size());
}

@Test
public void testRemove_Number() {
    set.add(10.0);
    int sizeBefore = set.size();
    set.remove(10.0);
    assertEquals(sizeBefore - 1, set.size());
}

You could write more tests, for example you could test your exceptions:

@Test(expected = IllegalArgumentException.class)
public void testAddingNaN() {
    set.add(Double.NaN);
}

Try to use a code coverage tool to see how much code your test cover and try to get a good percentage of coverage.