Support subfolders while importing Grafana Dashboards

[IvanZenger]

Is your feature request related to a problem? Please describe.
I would like to import the Grafana Harvest Dashboards into our Grafana instance under the folder NetApp/Harvest. However, when I use the command --serverfolder "NetApp/Harvest", it does not place the dashboards in the subfolder "Harvest" under "NetApp". Instead, it creates a new folder called NetApp/Harvest at the root level and imports the dashboards there.

Describe the solution you'd like
Enhance the grafana import command with a --folderuid flag that allows specifying the unique ID (UID) of the folder where the dashboards should be imported.

Describe alternatives you've considered
Improve the --serverfolder "NetApp/Harvest" option to recognize and handle recursive paths correctly.

[cgrinds]

hi @IvanZenger what version of Grafana are you using? It was only recently that Grafana added support for subfolders in Grafana 11.

My read of Grafana's create folder API indicates that for subfolders to work in Grafana you need to enable the nestedFolders feature flag in Grafana and looks like Harvest will need to pass a parentUid field when importing subfolders.

[IvanZenger]

hi @IvanZenger what version of Grafana are you using? It was only recently that Grafana added support for subfolders in Grafana 11.

My read of Grafana's create folder API indicates that for subfolders to work in Grafana you need to enable the nestedFolders feature flag in Grafana and looks like Harvest will need to pass a parentUid field when importing subfolders.

Hello @cgrinds,

We are using Grafana version 11.2.3. The FeatureToggle nestedFolder is enabled as default since v11.0 and was added as Beta-Feature in v10.0 in Jun 2023.

The parentUid is not required when a dashboard is imported or created. It is only needed if a folder must be created. As in the first proposed solution, using the --folderUid parameter, we specify that the folder already exists and does not need to be created. create/update dashboard

In the second solution, the implementation is quite open to interpretation. I would likely parse the path and recursively check if each folder exists. During this process, I would always know the folderUid of the parent folder, which allows me to use it as the parentUid for creating subfolders if they do not exist.

[cgrinds]

Thanks @IvanZenger! We'll add this

[IvanZenger]

Great. If I can help somewhere, just let me know.

Thanks a lot.

[cgrinds]

We opted for option 2 since it's more convenient to specify the folder names instead of forcing customers to lookup a folder's uid.

Here's an example:

bin/harvest grafana import --directory grafana/dashboards/cmode --serverfolder NetApp/Harvest

When running against a version of Grafana < 11.0.0, this will import like so:

When running against a version of Grafana >= 11.0.0, this will import like so:

[IvanZenger]

Very Cool @cgrinds,

I tested it this morning, and it works as described. However, I have a observation:

If I try to overwrite the dashboards, I receive a server response 409 on a POST call to /api/folders.

Command:
bin/grafana import --config harvest.yml --directory grafana/dashboards/cmode --serverfolder "NetApp/Harvest" --overwrite

Output:

using API token from config
connected to Grafana server (version: 11.2.3)
server response: 409 Conflict

If i change the Subfolder (example: "NetApp/Harvest 2.0") it works. The Folder "NetApp" is therefore not the problem. I guess the subfolder is not found and therefore it wants to create it.

harvest/cmd/tools/grafana/grafana.go

Lines 1221 to 1225 in 4832f74

	if err := checkFolder(curFolder); err != nil {
	return err
	}
	if curFolder.id == 0 {

[cgrinds]

Thanks for the feedback @IvanZenger. You're correct, Grafana's /api/folders API requires passing a parentUid= query param to retrieve subfolders. Commit 51b2dbf fixes that bug. Thanks for reporting!

Unrelated to this PR, overwrite was deprecated a few releases ago. When you use it, the following message is logged

warning: The overwrite flag is no longer used and will be removed in a future release. Please remove this flag from your command line invocation. When importing, dashboards are always overwritten.

[IvanZenger]

Very Cool. Thanks a lot. @cgrinds @Hardikl

[IvanZenger]

Do you know when this will be released?

[rahulguptajss]

Do you know when this will be released?

@IvanZenger These changes are available in the nightly build if you would like to try them out. They will also be included in the Harvest 25.02 release, scheduled for February.

[Hardikl]

Verified in main at commit 4b2cbc0,

Behaviour in Grafana 8.1.8,

harvest % bin/grafana import --config harvest.yml --directory grafana/dashboards/cmode --serverfolder "NetApp/Harvest"
using API token from config
connected to Grafana server (version: 8.1.8)
created Grafana folder [NetApp/Harvest] - OK
preparing to import dashboards...
OK - imported NetApp/Harvest / [aggregate.json]
OK - imported NetApp/Harvest / [cdot.json]
OK - imported NetApp/Harvest / [changelogMonitor.json]
OK - imported NetApp/Harvest / [cluster.json]
OK - imported NetApp/Harvest / [data_protection.json]
OK - imported NetApp/Harvest / [datacenter.json]
OK - imported NetApp/Harvest / [disk.json]
OK - imported NetApp/Harvest / [external_service_op.json]
OK - imported NetApp/Harvest / [flexcache.json]
OK - imported NetApp/Harvest / [flexgroup.json]
OK - imported NetApp/Harvest / [fsa.json]
OK - imported NetApp/Harvest / [headroom.json]
OK - imported NetApp/Harvest / [health.json]
OK - imported NetApp/Harvest / [lun.json]
OK - imported NetApp/Harvest / [mcc_cluster.json]
OK - imported NetApp/Harvest / [metadata.json]
OK - imported NetApp/Harvest / [namespace.json]
OK - imported NetApp/Harvest / [network.json]
OK - imported NetApp/Harvest / [nfs4storePool.json]
OK - imported NetApp/Harvest / [nfsTroubleshooting.json]
OK - imported NetApp/Harvest / [nfs_clients.json]
OK - imported NetApp/Harvest / [node.json]
OK - imported NetApp/Harvest / [power.json]
OK - imported NetApp/Harvest / [qtree.json]
OK - imported NetApp/Harvest / [quotaReport.json]
OK - imported NetApp/Harvest / [s3ObjectStorage.json]
OK - imported NetApp/Harvest / [security.json]
OK - imported NetApp/Harvest / [shelf.json]
OK - imported NetApp/Harvest / [smb.json]
OK - imported NetApp/Harvest / [snapmirror.json]
OK - imported NetApp/Harvest / [snapmirror_destinations.json]
OK - imported NetApp/Harvest / [svm.json]
OK - imported NetApp/Harvest / [volume.json]
OK - imported NetApp/Harvest / [vscan.json]
OK - imported NetApp/Harvest / [workload.json]
Imported 35 dashboards to [NetApp/Harvest] from [grafana/dashboards/cmode]
harvest % 

One new folder NetApp/Harvest at root level has been created.

Behaviour in Grafana 11.4.0,

harvest % bin/grafana import --config harvest.yml --directory grafana/dashboards/cmode --serverfolder "NetApp/Harvest"
using API token from config
error connect: (401 - 401 Unauthorized) Unauthorized
enter API token: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
save API key for later use? [Y/n]: y
saving config file [harvest.yml]
connected to Grafana server (version: 11.4.0)
created Grafana folder [NetApp/Harvest] - OK
preparing to import dashboards...
OK - imported Harvest / [aggregate.json]
OK - imported Harvest / [cdot.json]
OK - imported Harvest / [changelogMonitor.json]
OK - imported Harvest / [cluster.json]
OK - imported Harvest / [data_protection.json]
OK - imported Harvest / [datacenter.json]
OK - imported Harvest / [disk.json]
OK - imported Harvest / [external_service_op.json]
OK - imported Harvest / [flexcache.json]
OK - imported Harvest / [flexgroup.json]
OK - imported Harvest / [fsa.json]
OK - imported Harvest / [headroom.json]
OK - imported Harvest / [health.json]
OK - imported Harvest / [lun.json]
OK - imported Harvest / [mcc_cluster.json]
OK - imported Harvest / [metadata.json]
OK - imported Harvest / [namespace.json]
OK - imported Harvest / [network.json]
OK - imported Harvest / [nfs4storePool.json]
OK - imported Harvest / [nfsTroubleshooting.json]
OK - imported Harvest / [nfs_clients.json]
OK - imported Harvest / [node.json]
OK - imported Harvest / [power.json]
OK - imported Harvest / [qtree.json]
OK - imported Harvest / [quotaReport.json]
OK - imported Harvest / [s3ObjectStorage.json]
OK - imported Harvest / [security.json]
OK - imported Harvest / [shelf.json]
OK - imported Harvest / [smb.json]
OK - imported Harvest / [snapmirror.json]
OK - imported Harvest / [snapmirror_destinations.json]
OK - imported Harvest / [svm.json]
OK - imported Harvest / [volume.json]
OK - imported Harvest / [vscan.json]
OK - imported Harvest / [workload.json]
Imported 35 dashboards to [Harvest] from [grafana/dashboards/cmode]
harvest %