Filter Option or FlatMap in spark

Question

This is the code in Spark:

dsPhs.filter(filter(_))
.map(convert)
.coalesce(partitions)
.filter(additionalFilter.IsValid(_))

At convert function I get more complex object - MyObject, so I need to prefilter basic object. I have 3 options:

1. Make map return option(MyObject) and filter it at additionalFilter
2. Replace map with flatMap and return empty array when filtered
3. Use filter before map function, to filter out RawObject, before converting it to MyObject.

Answer 1

If, for option 2, you mean have convert return an empty array, there's another option: have convert return an Option[MyObject] and use flatMap instead of map. This has the best of options 1 and 2. Without knowing more about your use case, I can't say for sure whether this is better than option 3, but here are some considerations:

1. Should convert contain input validation logic? If so, consider modifying it to return an Option.
   • If convert is used, or will be used, in other places, could they benefit from this validation?
   • As a side note, this might be a good time to consider what convert currently does when passed an invalid argument.
2. Can you easily change convert and its signature? If not, consider using a filter.