Aggregate

If supportsPartial is true, there will be partialAggregate and finalAggregate. The former have requiredChildDistributionExpressions = None, indicating there will be no shuffle happen and the partial aggregation happening in place. But note that for SortAggregateExec, the requiredChildOrdering is setup with the groupingExpressions. The latter have some distribution requirements as requiredChildDistributionExpressions = Some(groupingAttributes). Then there will be ShuffleExchange inserted, and possibly Sort as well.

The aggregation is happening inside regular RDD, instead of inside of ShuffleDependency (aggregator inside of ShuffleDependency is None). As a result, there will be always BypassMergeSortShuffleWriter or UnsafeShuffleWriter used.

For aggregation without Distinct, it goes through regular process, with 1st mode as Partial, and 2nd mode as Final.

For aggregation with Distinct

SELECT COUNT(A), COUNT(DISTINCT(B)) FROM TABLE1 GROUP BY C

1st: change to SELECT COUNT(A) FROM TABLE1 GROUP BY B, C 1.1: use Partial to evalute COUNT(A), 1.2 use PartialMerge which requires shuffling because we have to do distinct through group by

2nd: change to "select count(a), count(b) from table1 group by c 2.1 create Partial distinct aggregate, and regular PartialMerge for count(a). Note that we have to rewrite B to be a attribute reference, as it has been evaluated in group by of 1 3.2.2 final aggregate