Keras structured SIMD pruning

[reuvenperetz]

Pull Request Description:

This pull request introduces an SIMD structured pruning to Keras models. The primary goal is to optimize models to meet specific Key Performance Indicators (KPIs). Key components include:

1. keras_pruning_experimental Function:

   • Interface to prune Keras models. It includes converting Keras models into an internal graph representation, applying compression configuration, and initiating the pruning process.
   • Integrates with Pruner class to apply graph pruning based on specified KPIs.

2. Pruner Class:

   • Central component responsible for executing the pruning process on the computational graph.
   • Computation of importance scores for each node using the provided data generator.
   • Supports pruning strategies with an initial focus on Greedy approach.

3. GreedyMaskCalculator Class:

   • Computes pruning masks for each prunable node in a Keras model's computational graph, using a greedy algorithm. It aims to meet a target KPI for memory footprint.
   • SIMD group indices and scores initialization for SIMD mask updates.
   • Incorporates a MemoryCalculator to estimate the memory footprint of the pruned graph.
   • get_mask(): Retrieves the pruning mask for the graph, computing it if not already done.

4. LFHImportanceMetric Class:

   • Implements the calculation of Label-Free-Hessian (LFH) based importance scores for nodes in the graph.
   • get_entry_node_to_score: Calculates importance scores for each entry node in a provided list using the LFH method. It then normalizes scores using L2 norms and the number of parameters for each output channel.

5. MemoryCalculator Class:

   • Estimation of the memory usage of a neural network's computational graph under various pruning masks.
   • get_pruned_graph_memory: Calculates the estimated memory usage of the pruned graph.
   • get_pruned_graph_num_params: Computes the total number of parameters in the pruned graph.

6. build_pruned_graph Function:

   • This function prunes a given computational graph based on specified pruning masks for output channels.
     It returns a new, pruned version of the computational graph.

7. PruningConfig Class:

   • Serves as a configuration class that specifies the approach and criteria for pruning a neural network.
   • num_score_approximations (int): Specifies the number of score approximations to be performed when calculating the importance of channels.
   • importance_metric (ImportanceMetric): Dictates the metric used to assess the importance of channels in the network. The default is set to Label-Free-Hessian (LFH) approximation.
   • channels_filtering_strategy (ChannelsFilteringStrategy): Determines the strategy for filtering or pruning channels, with the default strategy being a Greedy approach.

8. PruningSectionMask Class:

   • Represents the masks to be applied to a specific section of a neural network during pruning, including both input and output channels at the entry and exit nodes.

9. PruningSection Class:

   • Represents a section within a graph that is targeted for pruning, including an entry node, any intermediate nodes, and an exit node.

Checklist before requesting a review:

• I set the appropriate labels on the pull request.
• I have added/updated the release note draft (if necessary).
• I have updated the documentation to reflect my changes (if necessary).
• All function and files are well documented.
• All function and classes have type hints.
• There is a licenses in all file.
• The function and variable names are informative.
• I have checked for code duplications.
• I have added new unittest (if necessary).

[haihabi]

Missing description

[ofirgo]

still not sure what the answer is here