Tukukkk

My column col1 is an Array.

I know that col1.getItem(2) allows you to access the second argument of the column. Is there a function to access with argument as column col1.getItem(col2) ?

I can create a UDF but I would have to specify which type the array is (and it can be multiple type) so a generic way would be better and welcome !

The UDF I use:

  def retrieveByIndexSingle[T : ClassTag](value:Seq[T] ,index:Int,offset:Int=0):T = value(index + offset)



  def retrieveByIndexSingleDUDF = udf((value:Seq[Double] ,index:Int) => {

    retrieveByIndexSingle[Double](value, index)

  })



  def retrieveByIndexSingleSUDF = udf((value:Seq[String] ,index:Int) => {

    retrieveByIndexSingle[String](value, index)

  })

It is possible to use SQL expression for example with expr:

import org.apache.spark.sql.functions.expr



val df = Seq(

    (Seq("a", "b", "c"), 0), (Seq("d", "e", "f"), 2)

).toDF("col1", "col2")

df.withColumn("col3", expr("col1[col2]")).show

+---------+----+----+

|     col1|col2|col3|

+---------+----+----+

|[a, b, c]|   0|   a|

|[d, e, f]|   2|   f|

+---------+----+----+

or, in Spark 2.4 or later, element_at function:

import org.apache.spark.sql.functions.element_at



df.withColumn("col3", element_at($"col1", $"col2" + 1)).show

+---------+----+----+

|     col1|col2|col3|

+---------+----+----+

|[a, b, c]|   0|   a|

|[d, e, f]|   2|   f|

+---------+----+----+

Please note that at the moment (Spark 2.4) there is inconsistency between these two methods:

• SQL indexing is 0-based.


• 
  element_at indexing is 1-based.