What is the expected result when writing nullables on non-nullable attribute?

[cgiachalis]

When a schema has attributes that no nullables allowed, do we expect the write operation to succeed when the writing data include nullables? See what happens with df1 in the example:

library(tiledb)

createArray <- function(nullable = TRUE) {

  uri <- tempfile()


  domain <- tiledb_domain(tiledb_dim("row", c(0L, 100L), 100L, "INT32"))
  attrib <- c(tiledb_attr("int8",   type = "INT8",    nullable = nullable),
              tiledb_attr("int16",  type = "INT16",   nullable = nullable),
              tiledb_attr("int32",  type = "INT32",   nullable = nullable),
              tiledb_attr("int64",  type = "INT64",   nullable = nullable),
              tiledb_attr("float32",type = "FLOAT32", nullable = nullable),
              tiledb_attr("float64",type = "FLOAT64", nullable = nullable),
              tiledb_attr("uint8",  type = "UINT8",   nullable = nullable),
              tiledb_attr("uint16", type = "UINT16",  nullable = nullable),
              tiledb_attr("uint32", type = "UINT32",  nullable = nullable),
              tiledb_attr("uint64", type = "UINT64",  nullable = nullable))

  schema <- tiledb_array_schema(domain, attrib, sparse=TRUE)
  res <- tiledb_array_create(uri, schema)

  df <- data.frame(row     =  1:10,
                   int8    =  10L*c(1:2, NA, 4:10),
                   int16   =  20L*c(1:3, NA, 5:10),
                   int32   =  30L*c(1:4, NA, 6:10),
                   int64   =  bit64::as.integer64(40L*c(1:5, NA, 7:10)),
                   float32 =  50*c(1:6, NA, 8:10),
                   float64 =  60*c(1:7, NA, 9:10),
                   uint8   =  10*c(1:8, NA, 10),
                   uint16  =  80*c(1:9, NA),
                   uint32  =  90*c(1:8, NA, 10),
                   uint64  = bit64::as.integer64(100*c(1:7, NA, 9:10)))

  arr <- tiledb_array(uri, return_as="data.table")
  arr[] <- df
  arr
}

# no nullables
df1 <- createArray(nullable = FALSE)
df1[]
#>       row  int8 int16 int32 int64 float32 float64 uint8 uint16 uint32
#>     <int> <int> <int> <int> <i64>   <num>   <num> <int>  <int>  <int>
#>  1:     1    10    20    30    40      50      60    10     80     90
#>  2:     2    20    40    60    80     100     120    20    160    180
#>  3:     3     0    60    90   120     150     180    30    240    270
#>  4:     4    40     0   120   160     200     240    40    320    360
#>  5:     5    50   100    NA   200     250     300    50    400    450
#>  6:     6    60   120   180  <NA>     300     360    60    480    540
#>  7:     7    70   140   210   280     NaN     420    70    560    630
#>  8:     8    80   160   240   320     400      NA    80    640    720
#>  9:     9    90   180   270   360     450     540     0    720     NA
#> 10:    10   100   200   300   400     500     600   100      0    900
#>            uint64
#>             <num>
#>  1:  1.000000e+02
#>  2:  2.000000e+02
#>  3:  3.000000e+02
#>  4:  4.000000e+02
#>  5:  5.000000e+02
#>  6:  6.000000e+02
#>  7:  7.000000e+02
#>  8: -9.223372e+18
#>  9:  9.000000e+02
#> 10:  1.000000e+03

# nullables
df2 <- createArray(nullable = TRUE)
df2[]
#>       row  int8 int16 int32 int64 float32 float64 uint8 uint16 uint32 uint64
#>     <int> <int> <int> <int> <i64>   <num>   <num> <int>  <int>  <int>  <num>
#>  1:     1    10    20    30    40      50      60    10     80     90    100
#>  2:     2    20    40    60    80     100     120    20    160    180    200
#>  3:     3    NA    60    90   120     150     180    30    240    270    300
#>  4:     4    40    NA   120   160     200     240    40    320    360    400
#>  5:     5    50   100    NA   200     250     300    50    400    450    500
#>  6:     6    60   120   180  <NA>     300     360    60    480    540    600
#>  7:     7    70   140   210   280      NA     420    70    560    630    700
#>  8:     8    80   160   240   320     400      NA    80    640    720     NA
#>  9:     9    90   180   270   360     450     540    NA    720     NA    900
#> 10:    10   100   200   300   400     500     600   100     NA    900   1000

Passing nullable data to non-nullable attrs and reading back we could get NaN, NA , i64 NA, 0, .. depending on column type.

Would that behaviour defy the role of nullable argument in tiledb_attr()?

[johnkerl]

Some supporting information: given the arrays written by TileDB-R as above, reading them using TileDB-Py:

# nullable=false
   row  int8  int16       int32                int64  float32  float64  uint8  uint16      uint32               uint64
0    1    10     20          30                   40     50.0     60.0     10      80          90                  100
1    2    20     40          60                   80    100.0    120.0     20     160         180                  200
2    3     0     60          90                  120    150.0    180.0     30     240         270                  300
3    4    40      0         120                  160    200.0    240.0     40     320         360                  400
4    5    50    100 -2147483648                  200    250.0    300.0     50     400         450                  500
5    6    60    120         180 -9223372036854775808    300.0    360.0     60     480         540                  600
6    7    70    140         210                  280      NaN    420.0     70     560         630                  700
7    8    80    160         240                  320    400.0      NaN     80     640         720  9223372036854775808
8    9    90    180         270                  360    450.0    540.0      0     720  2147483648                  900
9   10   100    200         300                  400    500.0    600.0    100       0         900                 1000

# nullable=true
   row  int8  int16  int32  int64  float32  float64  uint8  uint16  uint32  uint64
0    1    10     20     30     40     50.0     60.0     10      80      90     100
1    2    20     40     60     80    100.0    120.0     20     160     180     200
2    3  <NA>     60     90    120    150.0    180.0     30     240     270     300
3    4    40   <NA>    120    160    200.0    240.0     40     320     360     400
4    5    50    100   <NA>    200    250.0    300.0     50     400     450     500
5    6    60    120    180   <NA>    300.0    360.0     60     480     540     600
6    7    70    140    210    280      NaN    420.0     70     560     630     700
7    8    80    160    240    320    400.0      NaN     80     640     720    <NA>
8    9    90    180    270    360    450.0    540.0   <NA>     720    <NA>     900
9   10   100    200    300    400    500.0    600.0    100    <NA>     900    1000

Notice that in the nullable=false case, the affected readback values are shown to be the fill values for each column's datatype. TileDB-Py shows these fill values without modification. TileDB-R, by contrast, is mapping these to R NA types. cc @mojaveazure .

[cgiachalis]

Driven by @johnkerl comment, is nullable argument meant for signalling whether the read back values
are filled by NA or not in the presence of missing values?

Let's start from nullable description in ?tiledb_attr:

nullable: (default FALSE) A logical switch whether the attribute can have missing values

Someone can infer that the default case doesn't allow missing values (NA). But it seems to happen this:

1. nullable = FALSE, set the read back fill values to default value (note you cant set all types by yourself with tiledb_attribute_set_fill_value)
2. nullable = TRUE , set the read back fill values to NA.

If that holds true, it's safer and more manageable for user to set always nullable = TRUE; and if NAs are not desired, then force the end user to sanitise them.