19-froc-sample-size.Rmd

# (PART\*) FROC sample size {-}


# FROC sample size estimation {#froc-sample-size}


```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
library(RJafroc)
library(ggplot2)
```


## How much finished 99 percent {#froc-sample-size-how-much-finished}

<!-- Comments on reason for increased FROC power; discuss idea about separating treatments and using lower half in terms of AUC to compute the NH parameters; does this improve DOBBINS-1 performance - solve the unusually low scale factor for this dataset? -->

## Overview

This chapter is split into two parts: 

* Part 1 is a step-by-step (or first-principles) approach to FROC paradigm sample size estimation. 

* Part 2 encapsulates some of the details in function `SsFrocNhRsmModel()` which makes it easier to use the sample size estimation method.


## Part 1

### Introduction {#froc-sample-size-intro}

FROC sample size estimation is not fundamentally different from ROC sample size estimation detailed in Chapter \@ref(roc-sample-size-dbm) and summarized next. 

**Summary of ROC sample size estimation** 

Based on analysis of a pilot ROC dataset and using a specified figure of merit (FOM), e.g., `FOM = Wilcoxon`, and either `method = "DBM"` or `method = "OR"` for significance testing, one estimates the intrinsic variability of the data expressed in terms of FOM variance components. For the DBM method these are the pseudovalue-based variance components while for OR method these are the FOM-based variance and covariances. **In this chapter the OR method will be used.** The second step is to specify a clinically realistic effect-size, e.g., the anticipated AUC difference between the two modalities. 

Given the variance components and the anticipated AUC difference the sample size functions (`RJafroc` function names beginning with `Ss`) allow one to estimate the number of readers and cases necessary to detect (i.e., reject the null hypothesis) the modality AUC difference with probability $\beta$, typically chosen to be 20 percent (corresponding to 80 percent statistical power) while maintaining the NH (zero AUC difference) rejection rate probability at $\alpha$, typically chosen to be 5 percent. 


**Summary of FROC sample size estimation** 

In FROC analysis the only difference, indeed the critical difference, is the choice of FOM; e.g., `FOM = "wAFROC"` instead of the ROC-AUC, `FOM = "Wilcoxon"`. The FROC dataset is analyzed using the OR method. This yields the covariance matrix corresponding to wAFROC-AUC FOM. Next one specifies the effect-size **in wAFROC-AUC units** and this step requires care. The ROC-AUC has a historically well-known interpretation, namely it is the classification ability at separating diseased patients from non-diseased patients, while the wAFROC-AUC does not. Needed is a way of relating the effect-size in the easily understood ROC-AUC unit to one in wAFROC-AUC unit. This requires a physical model, e.g., the RSM, that predicts both ROC and wAFROC curves and their corresponding AUCs.


1.	One chooses an ROC-AUC effect-size that is realistic and one that clinicians understand and can therefore participate in the effect-size postulation process. Lacking such information I recommend, based on past ROC studies, 0.03 as typical of a small effect size and 0.05 as typical of a moderate effect size.

2.	One converts the ROC effect-size to a wAFROC-AUC effect-size using the method described in the next section.

3.	One uses the sample size tools in `RJafroc` to determine sample size for a desired statistical power. 


>**It is important to recognize is that all quantities have to be in the same units**. When performing ROC analysis, everything (variance components and effect-size) has to be in units of the selected FOM, e.g., `FOM = "Wilcoxon"`. When performing wAFROC analysis, everything has to be in units of the wAFROC-AUC, i.e., `FOM = "wAFROC"`. The variance components and effect-size in wAFROC-AUC units will be different from their corresponding ROC counterparts. In particular, as shown next, a given ROC-AUC effect-size generally corresponds to a larger effect-size in wAFROC-AUC units. The reason for this is that the range over which wAFROC-AUC can vary, namely 0 to 1, which is twice the corresponding ROC-AUC range (0.5 to 1). For the same reason the wAFROC variance components also tend to be larger than the ROC variance components.


The next section explains the steps used to implement #2 above. 

### Relating ROC and wAFROC effect-sizes

The steps are illustrated using `dataset04`, a 5 treatment, 4 radiologist and 200 case FROC dataset [@zanca2009evaluation] acquired on a 5-point scale.

#### Extract NH treatments

If there are more than two treatments in the pilot dataset, as in `dataset04`, one extracts those treatments that represent "almost" null hypothesis data (in the sense of similar AUCs): 

```{r}
frocDataNH <- DfExtractDataset(dataset04, trts = c(1,2))
```

The preceding code extracts treatments 1 and 2 which were found [@zanca2009evaluation] to be "almost" equivalent (i.e., the NH could not be rejected for the wAFROC-AUC difference between these treatments). More than two almost NH treatments can be used if they have similar AUCs, as this will improve the stability of the procedure. However, the final sample size predictions are restricted to two treatments in the pivotal study. 

The next two steps are needed since **the RSM fits binned ROC data**. 

#### Convert the FROC NH data to ROC

If the original data is FROC one converts it to ROC:

```{r}
rocDataNH <- DfFroc2Roc(frocDataNH)
```

#### Bin the data

If the NH dataset uses continuous ratings one bins the ratings: 

```{r}
# For dataset04 this is unnecessary as it is already binned, but it can't hurt, i.e.,
# rocDataBinNH <- rocDataNH # gives same results as following line
rocDataBinNH <- DfBinDataset(rocDataNH, opChType = "ROC")
```

The default number of bins should be used. Unlike binning using arbitrarily set thresholds the thresholds found by `DfBinDataset()` are unique as they are chosen to maximize the empirical ROC-AUC. 


#### Determine the lesion distribution and weights of the FROC dataset

```{r, echo=FALSE}
lesDistr <- UtilLesDistr(frocDataNH)
W <- UtilLesWghtsDS(frocDataNH)
```

`lesDistr` is the lesion distribution, see Section \@ref(quick-start-froc-data-lesion-distribution) and line 1 of the following code. The RSM fitting algorithm needs to know how lesion-rich the dataset is as the predicted ROC-AUC depends on it. For this dataset fraction `r lesDistr$Freq[1]` of diseased cases have one lesion, fraction `r lesDistr$Freq[2]` have two lesions and fraction `r lesDistr$Freq[3]` have three lesions. One also needs the lesion weights matrix, $\textbf{W}$, see Section \@ref(quick-start-froc-data-lesion-weights). The call at line 2 to `UtilLesWghtsDS` uses the default argument `relWeights = 0` which assigns equal weights to all lesions.

```{r, attr.source = ".numberLines"}
lesDistr <- UtilLesDistr(frocDataNH)
W <- UtilLesWghtsDS(frocDataNH)
```

Note that `lesDistr` and $\textbf{W}$ are determined from the **FROC** NH dataset as this information is lost upon conversion to an ROC dataset. 

#### Fit the RSM to the ROC data

For each treatment and reader the fitting algorithm `FitRsmRoc()` is applied (see below, lines 4 - 11) to the binned NH ROC dataset. The returned values are `mu`, `lambda` and `nu`, corresponding to the physical RSM parameters $\mu$, $\lambda$ and $\nu$.

```{r, attr.source = ".numberLines", cache=FALSE}
I <- dim(rocDataBinNH$ratings$NL)[1] # number of levels of treatment factor
J <- dim(rocDataBinNH$ratings$NL)[2] # number of levels of reader factor
RsmParmsNH <- array(dim = c(I,J,3)) # 3 corresponds to the three RSM parameters
for (i in 1:I) {
  for (j in 1:J)  {
    fit <- FitRsmRoc(rocDataBinNH, trt = i, rdr = j, lesDistr$Freq)
    RsmParmsNH[i,j,1] <- fit[[1]] # mu
    RsmParmsNH[i,j,2] <- fit[[2]] # lambda
    RsmParmsNH[i,j,3] <- fit[[3]] # nu
  }
}
```


#### Compute the median values of the RSM parameters

I recommend taking the median (not the mean) of the $I \times J$ values of each parameter as representing the average NH dataset (he median is less sensitive to outliers than the mean).  


```{r}
muNH <- median(RsmParmsNH[,,1]) 
lambdaNH <- median(RsmParmsNH[,,2])
nuNH <- median(RsmParmsNH[,,3])
```


The defining values of the RSM-based NH fitting model are `muNH` = `r muNH`, `lambdaNH` = `r lambdaNH` and `nuNH` = `r nuNH`. 


#### Compute ROC and wAFROC NH AUCs

The next step is the compute the analytical (i.e., RSM-based) AUCs under the respective ROC and the wAFROC curves:

```{r}
aucRocNH <- UtilAnalyticalAucsRSM(muNH, lambdaNH, nuNH, lesDistr = lesDistr$Freq)$aucROC
aucwAfrocNH <- UtilAnalyticalAucsRSM(muNH, lambdaNH, nuNH, lesDistr = lesDistr$Freq)$aucwAFROC
```


The AUCs are: `aucRocNH = `r aucRocNH`` and `aucwAfrocNH = `r aucwAfrocNH``. Note that the wAFROC-FOM is smaller than the ROC-FOM as it includes both detection and localization performance (the ROC-AUC only measures detection performance). 

#### Compute ROC and wAFROC Alternative Hypotheses AUCs for a range of ROC-AUC effect sizes

To create the alternative hypothesis (AH) condition, one increments $\mu_{NH}$ by $\Delta\mu$. Tempting as it may be, it is not enough to simply increase the $\mu$ parameter as increasing $\mu$ will simultaneously decrease $\lambda$ and increase $\nu$ (see the Astronomical Analogy in the [RJafrocFrocBook](https://dpc10ster.github.io/RJafrocFrocBook/froc-paradigm.html#froc-paradigm-solar-analogy)). To account for this tandem effect one extracts the **intrinsic** parameters $\lambda_i, \nu_i$, at line 7 of the following code, and then converts back to the physical parameters at line 9 using the incremented $\mu$. Note the usage of the functions `Util2Intrinsic` (convert physical to intrinsic) and `Util2Physical` (convert intrinsic to physical) . The resulting ROC-AUC and wAFROC-AUC are then calculated. This yields the effect size (AH value minus NH value) using ROC and wAFROC FOMs for a series of specified $\Delta\mu$ values. These are used to relate the wAFROC effect size for a specified ROC effect size. 


```{r, attr.source = ".numberLines"}
deltaMu <- seq(0.01, 0.2, 0.01)
esROC <- array(dim = length(deltaMu))
eswAFROC <- array(dim = length(deltaMu))

# get intrinsic parameters
par_i <- Util2Intrinsic(muNH, lambdaNH, nuNH) # convert physical to intrinsic

for (i in 1:length(deltaMu)) {
  
  # find physical parameters for the increased muNH (accounting for the tandem effects)
  par_p <- Util2Physical(muNH + deltaMu[i], par_i$lambda_i, par_i$nu_i)  # convert intrinsic to physical
  
  # AH ROC value minus NH ROC value
  esROC[i] <- UtilAnalyticalAucsRSM(
    muNH + deltaMu[i], par_p$lambda, par_p$nu, lesDistr = lesDistr$Freq)$aucROC - aucRocNH 
  
  # AH wAFROC value minus NH wAFROC value
  eswAFROC[i] <- UtilAnalyticalAucsRSM(
    muNH + deltaMu[i], par_p$lambda, par_p$nu, lesDistr = lesDistr$Freq)$aucwAFROC - aucwAfrocNH 
  
}
```


Here is a plot of wAFROC effect size (y-axis) vs. ROC effect size.


```{r, echo = FALSE, fig.show='hold', fig.align='center', fig.cap="Plot of wAFROC effect size vs. ROC effect size. The straight line fit through the origin has slope 2.169."}
df <- data.frame(esROC = esROC, eswAFROC = eswAFROC)
p <- ggplot(data = df, aes(x = esROC, y = eswAFROC)) +
  geom_smooth(method = "lm", se = FALSE, color = "black", formula = y ~ x) +
  geom_point(size = 4) +
  scale_color_manual(values = "black") + 
  theme(axis.title.y = element_text(size = 10,face="bold"),
        axis.title.x = element_text(size = 10,face="bold")) +
  scale_x_continuous(expand = c(0, 0)) + 
  scale_y_continuous(expand = c(0, 0)) 
print(p)
```             


The plot is linear and the intercept is close to zero. This makes it easy to implement an interpolation function. In the following code line 1 fits `eswAFROC` vs. `esROC` using a linear model `lm()` function constrained to pass through the origin (the "-1"). One expects this constraint since for `deltaMu = 0` the effect size must be zero no matter how it is measured. 

```{r, attr.source = ".numberLines"}
lmFit <- lm(eswAFROC ~ -1 + esROC) # the "-1" fits to straight line through the origin
scaleFactor <- lmFit$coefficients
effectSizeROC <- seq(0.01, 0.0525, 0.0025) 
effectSizewAFROC <- effectSizeROC*scaleFactor
```


The slope of the zero-intercept constrained straight line fit is `scaleFactor` = `r round(scaleFactor,3)` and the squared correlation coefficient is `R2` = `r summary(lmFit)$r.squared` (the fit is very good). Therefore, the conversion from ROC to wAFROC effect size is: 

```
effectSizewAFROC = scaleFactor * effectSizeROC
```

**For this dataset the wAFROC effect size is 2.169 times the ROC effect size.** The wAFROC effect size is expected to be larger than the ROC effect size because the range of wAFROC-AUC, $1-0=1$, is twice that of ROC-AUC, $1-0.5=0.5$.


### ROC and wAFROC variance components

The following skeleton code shows the arguments of the function `UtilORVarComp` used to calculate the OR variance components (other arguments are left at their default values).

```
UtilORVarComp(
  dataset,
  FOM
)
```

`UtilORVarComp()` is applied to `rocDataNH` and `frocDataNH` (using "Wilcoxon" and "wAFROC" FOMs as appropriate) followed by the extraction of the ROC and wAFROC variance components.


```{r, attr.source = ".numberLines"}
varComp_roc <- UtilORVarComp(
  rocDataNH, 
  FOM = "Wilcoxon")$VarCom[-2]

varComp_wafroc <- UtilORVarComp(
  frocDataNH, 
  FOM = "wAFROC")$VarCom[-2]
```


VarCom[-2] removes the second column of each dataframe containing the correlations. The ROC and wAFROC variance components are:


```{r, echo=FALSE}
varComp_both <- cbind(varComp_roc, varComp_wafroc)
colnames(varComp_both) <-  c("ROC", "wAFROC")
varComp_both
```



### ROC and wAFROC power for equivalent effect-sizes

The following code compares ROC and wAFROC random-reader random-case (RRRC) powers for equivalent effect sizes.  

First, one needs the numbers of readers `JStar` and cases `KStar` in the pilot dataset (lines 1 - 2 in following code) and those in the pivotal study, `JPivot` and `KPivot` (line 3). The values for the pivotal study have been arbitrarily set at 5 readers and 100 cases. 

```{r, attr.source = ".numberLines"}
JStar <- length(dataset04$ratings$NL[1,,1,1])
KStar <- length(dataset04$ratings$NL[1,1,,1])
JPivot <- 5;KPivot <- 100
```

Next one extracts the OR ROC variance components from the previously computed list `varComp_roc`.

```{r, attr.source = ".numberLines"}
# these are OR variance components assuming FOM = "Wilcoxon"
varR_roc <- varComp_roc["VarR","Estimates"]
varTR_roc <- varComp_roc["VarTR","Estimates"]
Cov1_roc <- varComp_roc["Cov1","Estimates"]
Cov2_roc <- varComp_roc["Cov2","Estimates"]
Cov3_roc <- varComp_roc["Cov3","Estimates"]
Var_roc <- varComp_roc["Var","Estimates"]
```

The procedure is repeated for the OR wAFROC variance components using the previously computed list `varComp_wafroc`.

```{r, attr.source = ".numberLines"}
# these are OR variance components assuming FOM = "wAFROC"
varR_wafroc <- varComp_wafroc["VarR","Estimates"]
varTR_wafroc <- varComp_wafroc["VarTR","Estimates"]
Cov1_wafroc <- varComp_wafroc["Cov1","Estimates"]
Cov2_wafroc <- varComp_wafroc["Cov2","Estimates"]
Cov3_wafroc <- varComp_wafroc["Cov3","Estimates"]
Var_wafroc <- varComp_wafroc["Var","Estimates"]
```

We are now ready for the power calculations. The needed function is `SsPowerGivenJK` ("sample size for given number of readers J and cases K"):

```
SsPowerGivenJK(
  dataset,
  FOM,
  J,
  K,
  effectSize,
  ...
)
```

In the following code the OR ROC variance components are passed to `SsPowerGivenJK` at lines 12-18. The OR wAFROC variance components are passed to `SsPowerGivenJK` at lines 29-34. Setting `dataset = NULL` means that the function does not need a dataset as the variance components are supplied instead using the `...` argument.

The for-loop (lines 3 - 36) calculates ROC power (line 19) and wAFROC power (line 35) for a number of ROC (line 11) and corresponding wAFROC (line 28) effect sizes.


```{r, attr.source = ".numberLines"}
power_roc <- array(dim = length(effectSizeROC))
power_wafroc <- array(dim = length(effectSizeROC))
for (i in 1:length(effectSizeROC)) {
  # compute ROC power
  # dataset = NULL means use the supplied variance components instead of dataset
  ret <- SsPowerGivenJK(
    dataset = NULL, 
    FOM = "Wilcoxon", 
    J = JPivot, 
    K = KPivot, 
    effectSize = effectSizeROC[i], 
    list(JStar = JStar, 
         KStar = KStar, 
         VarTR = varTR_roc,
         Cov1 = Cov1_roc,
         Cov2 = Cov2_roc,
         Cov3 = Cov3_roc,
         Var = Var_roc))
  power_roc[i] <- ret$powerRRRC
  
  # compute wAFROC power
  # dataset = NULL means use the supplied variance components instead of dataset
  ret <- SsPowerGivenJK(
    dataset = NULL, 
    FOM = "wAFROC", 
    J = JPivot, 
    K = KPivot, 
    effectSize = effectSizewAFROC[i], 
    list(JStar = JStar, 
         KStar = KStar, 
         VarTR = varTR_wafroc,
         Cov1 = Cov1_wafroc,
         Cov2 = Cov2_wafroc,
         Cov3 = Cov3_wafroc,
         Var = Var_wafroc))
  power_wafroc[i] <- ret$powerRRRC
}
```


Since the wAFROC effect size is `r scaleFactor` times the ROC effect size, wAFROC power is larger than ROC power. For example, for ROC effect size = `r round(effectSizeROC[11],3)` the wAFROC effect size is `r round(effectSizewAFROC[11],3)`, the ROC power is `r round(power_roc[11],3)` while the wAFROC power is `r round(power_wafroc[11],3)`. The influence of the increased wAFROC effect size is magnified as it enters as the square in the formula for statistical power: this overwhelms the increase, noted previously, in variability of wAFROC-AUC relative to ROC-AUC 

The following is a plot of wAFROC power vs. ROC power for the specified effect sizes.


```{r, echo=FALSE, fig.show='hold', fig.align='center', fig.cap="Plot of wAFROC power vs. ROC power. For ROC effect size = 0.035 the wAFROC effect size is 0.0759, the ROC power is 0.234 while the wAFROC power is 0.796."}
df <- data.frame(power_ROC = power_roc, power_wAFROC = power_wafroc)
p <- ggplot(mapping = aes(x = power_ROC, y = power_wAFROC)) +
  geom_line(data = df, linewidth = 2)+
  scale_color_manual(values = "black") + 
  theme(axis.title.y = element_text(size = 10,face="bold"),
        axis.title.x = element_text(size = 10,face="bold"))#  +
print(p)
```             





## Part 2

### Introduction

This example uses the FED dataset as a pilot FROC study and function `SsFrocNhRsmModel()` (RSM-based FROC NH model) to construct the NH model (thereby encapsulating some of the code in the first part).


### Constructing the NH model

The first two treatments are extracted from `dataset04` thereby yielding the NH dataset (line 1). The lesion distribution is specified in line 2. `lesDistr` can be specified independent of that in the pilot dataset. This allows some control over selection of the diseased cases in the pivotal study. However, in this example it is simply extracted from the pilot dataset. Line 3 constructs the NH model using function `SsFrocNhRsmModel` to calculate the NH RSM parameters (lines 4 - 6) and the scale factor (line 7).

```{r, attr.source = ".numberLines"} 
frocNhData <- DfExtractDataset(dataset04, trts = c(1,2))
lesDistr <- UtilLesDistr(frocNhData) # this can be replaced by the anticipated lesion distribution
ret <- SsFrocNhRsmModel(frocNhData, lesDistr = lesDistr$Freq)
muNH <- ret$mu
lambdaNH <- ret$lambda
nuNH <- ret$nu
scaleFactor <- ret$scaleFactor
```


The fitting model is defined by `muNH` = `r muNH`,  `lambdaNH` = `r lambdaNH`  and  `nuNH` = `r nuNH` and `lesDistr$Freq` = `r lesDistr$Freq`. The effect size scale factor is `scaleFactor` = `r scaleFactor`. All of these are identical to the Part I values.


### Extract the wAFROC variance components  {#froc-sample-size-variance-components}

The code applies the significance testing function `St()` to `frocNhData`, using `FOM = "wAFROC"` and extracts the variance components.

```{r}
varComp_wafroc  <- St(
  frocNhData, 
  FOM = "wAFROC", 
  method = "OR", 
  analysisOption = "RRRC")$ANOVA$VarCom
```


### wAFROC power for specified ROC effect size, number of readers and number of cases

The following example is for ROC effect size = 0.035 (line 1), 5 readers and 100 cases (line 4) in the **pivotal study**. The function `SsPowerGivenJK` returns the power for the specified number of readers J (line 9), cases K (line 10) and wAFROC effect size (line 11). Since `dataset` is set to `NULL` JStar and KStar (corresponding to the pilot study) and the variance components are supplied as a `list` variable, lines 12 - 18. If dataset is specified then these are calculated from the pilot study dataset.  

```{r, attr.source = ".numberLines"}
effectSizeROC <- 0.035
effectSizewAFROC <- scaleFactor * effectSizeROC

J <- 5;K <- 100 # define pivotal study sample size

ret <- SsPowerGivenJK(
  dataset = NULL, # must set 
  FOM = "wAFROC", 
  J = J, 
  K = K, 
  effectSize = effectSizewAFROC, 
  list(JStar = JStar, 
       KStar = KStar, 
       VarTR = varTR_wafroc,
       Cov1 = Cov1_wafroc,
       Cov2 = Cov2_wafroc,
       Cov3 = Cov3_wafroc,
       Var = Var_wafroc))
power_wafroc <- ret$powerRRRC
```

```{r, echo=FALSE}
cat("ROC-ES = ", effectSizeROC, 
    ", wAFROC-ES = ", effectSizeROC * scaleFactor, 
    ", Power-wAFROC = ", power_wafroc, "\n")
```

### Number of cases for 80 percent power for a given number of readers

Function `SsSampleSizeKGivenJ` (number of cases K for desired power for given number of readers J) is shown below. If `dataset` is set to `NULL` then JStar and KStar and the variance components must be specified as a `list` variable, otherwise these are computed from `dataset`.

```
SsSampleSizeKGivenJ(
  dataset,
  ...,
  J,
  FOM,
  effectSize = NULL,
  alpha = 0.05,
  desiredPower = 0.8,
)
```

The following code returns the number of cases needed for 80 percent power for 6 readers (line 3), wAFROC effect size (line 4) = `r round(effectSizewAFROC, 3)` and JStar and KStar and wAFROC variance components (lines 5 -11). 


```{r, attr.source = ".numberLines"}
ret2 <- SsSampleSizeKGivenJ(
  dataset = NULL, 
  J = 6, 
  effectSize = effectSizewAFROC, 
  list(JStar = JStar, 
       KStar = KStar, 
       VarTR = varTR_wafroc,
       Cov1 = Cov1_wafroc,
       Cov2 = Cov2_wafroc,
       Cov3 = Cov3_wafroc,
       Var = Var_wafroc))
```


```{r, echo=FALSE}
cat("ROC-ES = ", effectSizeROC, 
    ", wAFROC-ES = ", effectSizeROC * scaleFactor, 
    ", K80RRRC = ", ret2$KRRRC, 
    ", Power-wAFROC = ", ret2$powerRRRC, "\n")
```


Here `K80RRRC` is the number of cases needed for 80 percent power when using RRRC analysis.

<!-- ## Part 3 -->


<!-- What follows is an application of the method to all my available FROC datasets. If more than two treatments are present two analyses are reported: one using all treatments and one using NH not-rejecting (NH-NR) treatments only. Note that ROC effect sizes were sometimes varied between the two analyses to keep the FROC power close to 80 percent. In all reported analyses J = 5 readers and K = 100 cases are assumed in the pivotal study.  -->



<!-- ### TONY dataset -->

<!-- This dataset has only two treatments; therefore a single analyses using both treatments is reported. -->


<!-- ```{r, echo=FALSE, cache=FALSE} -->
<!-- ds <- dataset01 -->
<!-- cat("dataset = ", ds$descriptions$fileName,  -->
<!-- ", Name = ", ds$descriptions$name,  -->
<!-- ", I = ", length(ds$ratings$NL[,1,1,1]),  -->
<!-- ", J = ", length(ds$ratings$NL[1,,1,1]),  -->
<!-- "\n\n") -->
<!-- effectSizeROC <- 0.07 -->
<!-- SsFrocSampleSize(ds, J = 5, K = 100, effectSizeROC = effectSizeROC) -->
<!-- ``` -->

<!-- ### FED dataset, all data and NH-NR treatments only -->

<!-- This dataset has five treatments; therefore two results are reported: one using all treatments and one using NH not-rejecting treatments only, i.e., `DfExtractDataset(ds, trts = c(1,2))`.  -->

<!-- ```{r, echo=FALSE, cache=FALSE} -->
<!-- ds <- dataset04 -->
<!-- cat("dataset = ", ds$descriptions$fileName,  -->
<!-- ", Name = ", ds$descriptions$name,  -->
<!-- ", I = ", length(ds$ratings$NL[,1,1,1]),  -->
<!-- ", J = ", length(ds$ratings$NL[1,,1,1]),  -->
<!-- "\n\n") -->

<!-- cat("All data", "\n\n") -->
<!-- effectSizeROC <- 0.04 -->
<!-- SsFrocSampleSize(ds, J = 5, K = 100, effectSizeROC = effectSizeROC) -->

<!-- cat("Note change in effect size in next analysis\n") -->
<!-- effectSizeROC <- 0.035 -->
<!-- cat("NH-NR treatments only: trts = c(1,2)", "\n\n") -->
<!-- SsFrocSampleSize(DfExtractDataset(ds, trts = c(1,2)), J = 5, K = 100, effectSizeROC = effectSizeROC) -->
<!-- ``` -->


<!-- ### THOMPSON dataset -->

<!-- This dataset has only two treatments; therefore a single analyses using both treatments is reported. -->


<!-- ```{r, echo=FALSE, cache=FALSE} -->
<!-- ds <- dataset05 -->
<!-- cat("dataset = ", ds$descriptions$fileName,  -->
<!-- ", Name = ", ds$descriptions$name,  -->
<!-- ", I = ", length(ds$ratings$NL[,1,1,1]),  -->
<!-- ", J = ", length(ds$ratings$NL[1,,1,1]),  -->
<!-- "\n\n") -->
<!-- effectSizeROC <- 0.05 -->
<!-- SsFrocSampleSize(ds, J = 5, K = 100, effectSizeROC = effectSizeROC) -->
<!-- ``` -->


<!-- ### MAGNUS dataset -->

<!-- This dataset has only two treatments; therefore a single analyses using both treatments is reported. -->


<!-- ```{r, echo=FALSE, cache=FALSE} -->
<!-- ds <- dataset06 -->
<!-- cat("dataset = ", ds$descriptions$fileName,  -->
<!-- ", Name = ", ds$descriptions$name,  -->
<!-- ", I = ", length(ds$ratings$NL[,1,1,1]),  -->
<!-- ", J = ", length(ds$ratings$NL[1,,1,1]),  -->
<!-- "\n\n") -->
<!-- effectSizeROC <- 0.05 -->
<!-- SsFrocSampleSize(ds, J = 5, K = 100, effectSizeROC = effectSizeROC) -->
<!-- ``` -->

<!-- ### LUCY dataset, all data and NH-NR treatments only -->

<!-- This dataset has five treatments; therefore two results are reported: one using all treatments and one using NH not-rejecting treatments only, i.e., `DfExtractDataset(ds, trts = c(1,5))`.  -->

<!-- ```{r, echo=FALSE, cache=FALSE} -->
<!-- ds <- dataset07 -->
<!-- cat("dataset = ", ds$descriptions$fileName,  -->
<!-- ", Name = ", ds$descriptions$name,  -->
<!-- ", I = ", length(ds$ratings$NL[,1,1,1]),  -->
<!-- ", J = ", length(ds$ratings$NL[1,,1,1]),  -->
<!-- "\n\n") -->

<!-- cat("All data", "\n") -->
<!-- effectSizeROC <- 0.06 -->
<!-- SsFrocSampleSize(ds, J = 5, K = 100, effectSizeROC = effectSizeROC) -->

<!-- cat("Note change in effect size in next analysis\n") -->
<!-- effectSizeROC <- 0.04 -->
<!-- cat("NH-NR treatments only: trts = c(1,5)", "\n") -->
<!-- SsFrocSampleSize(DfExtractDataset(ds, trts = c(1,5)), J = 5, K = 100, effectSizeROC = effectSizeROC) -->
<!-- ``` -->

<!-- ### DOBBINS-1 dataset, all data and NH-NR treatments only -->

<!-- Note: this dataset revealed issues with the RSM fitting program, namely the number of ROC operating points sometimes falls below 3, the minimum required.  -->

<!-- ```{r froc-sample-size-dataset11, echo=FALSE,out.width="50%",out.height="20%",fig.cap="RSM fitted ROC plot for dataset11, treatment 1 and reader 3. Note the two defining operating points near the origin. ",fig.show='hold',fig.align='center'} -->
<!-- knitr::include_graphics("images/froc-sample-size/ds11trt1rdr3.png") -->
<!-- ``` -->


<!-- This dataset has four treatments; therefore two results are reported: one using all treatments and one using NH not-rejecting treatments only, i.e., `DfExtractDataset(ds, trts = c(1,3,4))`.  -->

<!-- ```{r, echo=FALSE, cache=FALSE} -->

<!-- ds <- dataset11 -->
<!-- cat("dataset = ", ds$descriptions$fileName,  -->
<!-- ", Name = ", ds$descriptions$name,  -->
<!-- ", I = ", length(ds$ratings$NL[,1,1,1]),  -->
<!-- ", J = ", length(ds$ratings$NL[1,,1,1]),  -->
<!-- "\n\n") -->

<!-- cat("All data", "\n") -->
<!-- effectSizeROC <- 0.07 -->
<!-- SsFrocSampleSize(ds, J = 5, K = 100, effectSizeROC = effectSizeROC) -->

<!-- cat("NH-NR treatments only: trts = c(1,3,4)", "\n") -->
<!-- SsFrocSampleSize(DfExtractDataset(ds, trts = c(1,3,4)), J = 5, K = 100, effectSizeROC = effectSizeROC) -->
<!-- ``` -->

<!-- ### DOBBINS-3 dataset, all data and NH-NR treatments only -->

<!-- This dataset has four treatments; therefore two results are reported: one using all treatments and one using NH not-rejecting treatments only, i.e., `DfExtractDataset(ds, trts = c(3,4))`.  -->


<!-- ```{r, echo=FALSE, cache=FALSE} -->
<!-- ds <- dataset13 -->
<!-- cat("dataset = ", ds$descriptions$fileName,  -->
<!-- ", Name = ", ds$descriptions$name,  -->
<!-- ", I = ", length(ds$ratings$NL[,1,1,1]),  -->
<!-- ", J = ", length(ds$ratings$NL[1,,1,1]),  -->
<!-- "\n\n") -->

<!-- cat("All data", "\n") -->
<!-- effectSizeROC <- 0.07 -->
<!-- SsFrocSampleSize(ds, J = 5, K = 100, effectSizeROC = effectSizeROC) -->

<!-- cat("Note change in effect size in next analysis\n") -->
<!-- effectSizeROC <- 0.03 -->
<!-- cat("NH-NR treatments only: trts = c(3,4)", "\n") -->
<!-- SsFrocSampleSize(DfExtractDataset(ds, trts = c(3,4)), J = 5, K = 100, effectSizeROC = effectSizeROC) -->
<!-- ``` -->



<!-- With one exception, namely `dataset11`, each analyses demonstrates the considerable power advantage of FROC over ROC analysis. This is due to the increased predicted wAFROC-AUC effect size relative to the postulated ROC-AUC effect size. -->