Unable to use the tuning service with a week of data

[litan-shamir]

Describe the bug
We made load tests with a seasonal period of 240 (an hour) and it works great, better than expected.
But when modifying it to 5760 (a day) and a season of between 1-4, the tuning service returns 414 - Url too long.

Predictive Horizontal Pod Autoscaler Version
v0.13.0

To Reproduce
Steps to reproduce the behavior:

1. Install predictive-horizontal-pod-autoscaler crd
2. Set the PredictiveHorizontalPodAutoscaler manifest like so:

apiVersion: jamiethompson.me/v1alpha1
kind: PredictiveHorizontalPodAutoscaler
metadata:
  labels:
    app.kubernetes.io/instance: <my-pod-instance>
  name: <my-pod-instance>
  namespace: default
spec:
  behavior:
    scaleDown:
      policies:
        - periodSeconds: 330
          type: Percent
          value: 10
      stabilizationWindowSeconds: 300
  maxReplicas: 200
  metrics:
    - resource:
        name: cpu
        target:
          averageUtilization: 40
          type: Utilization
      type: Resource
  minReplicas: 40
  models:
    - holtWinters:
        alpha: 0.9
        beta: 0.9
        gamma: 0.9
        runtimeTuningFetchHook:
          http:
            method: GET
            parameterMode: query
            successCodes:
              - 200
            url: 'http://tuning.default.svc.cluster.local/holt_winters'
          timeout: 2500
          type: http
        seasonal: additive
        seasonalPeriods: 240
        storedSeasons: 3
        trend: additive
      name: HoltWintersPrediction
      type: HoltWinters
  scaleTargetRef:
    apiVersion: argoproj.io/v1alpha1
    kind: Rollout
    name: <my-pod-instance>
  syncPeriod: 30000

3. Install the helm chart containing the deployment and the tuning service
4. See the 414 - URL too long error code

Expected behavior
Expected the tuning service to make it's magic upon a week of data (to implement it into production). Instead it only gets an hour worth of data.

Kubernetes Details (kubectl version):
v1.22

Additional context
We need this feature because we have large peaks (from 20 pods to 100) at a specific time.

[jthomperoo]

Hi @litan-shamir, sorry for the super slow response!

It looks like the URL being generated to fetch the tuning information is too long, this can easily happen because it's all being sent as a query parameter, you don't need too much data to bump into this as an issue.

Rather than sending all of that data as a query parameter, it's probably better to send it as a body parameter (see the example in the docs).

So instead of:

parameterMode: query

You would use:

parameterMode: body

This would mean the tuning service would need to be updated to parse the body rather than the query parameter, but this should not be so limited by size as sending it as a query parameter.

I'll update the example included in the repository to use the body parameter over the query parameter, because I think in general it's a much better approach.

Thanks for raising this, hopefully this resolves your issue.

[jthomperoo]

I've updated the docs + the example to use the body parameter over query parameter.

Docs: https://predictive-horizontal-pod-autoscaler.readthedocs.io/en/latest/user-guide/models/#holt-winters-runtime-tuning
Example: https://github.com/jthomperoo/predictive-horizontal-pod-autoscaler/tree/f68184460ce6cdfdfa637d0e0e322151de8c833c/examples/dynamic-holt-winters