How to enable backing store for plots from UVCDAT script?

[cameronsmith1]

I am putting together a UVCDAT plotting script. Each time I plot I get a new window (which is good), but they have no backing store, and don't get redrawn. This is a problem because I am producing several plots, so have to move them around my screen, and there are so many they start covering each other, and when I bring a window to the front the part that was covered stays blank (ie white).

Below is an example (which is almost identical to the code provided by the UVCDAT GUI)

I am using UVCDAT v2.2 on Rhea.

Thanks!

canvas = vcs.init()
gmBoxfill = vcs.getboxfill('a_boxfill')
args = []
args.append(CLOUD)
gmBoxfill.boxfill_type = 'linear'
gmBoxfill.color_1 = 16
gmBoxfill.color_2 = 239
gmBoxfill.datawc_calendar = 135441
gmBoxfill.datawc_timeunits = 'days since 2000'
gmBoxfill.datawc_x1 = 1e+20
gmBoxfill.datawc_x2 = 1e+20
gmBoxfill.datawc_y1 = 1e+20
gmBoxfill.datawc_y2 = 1e+20
gmBoxfill.levels = [1e+20, 1e+20]
gmBoxfill.ext_1 = False
gmBoxfill.ext_2 = False
gmBoxfill.fillareacolors = None
gmBoxfill.fillareaindices = None
gmBoxfill.fillareastyle = 'solid'
gmBoxfill.legend = None
gmBoxfill.level_1 = 1e+20
gmBoxfill.level_2 = 1e+20
gmBoxfill.missing = 1
gmBoxfill.projection = 'linear'
gmBoxfill.xaxisconvert = 'linear'
gmBoxfill.xmtics1 = ''
gmBoxfill.xmtics2 = ''
gmBoxfill.xticlabels1 = '*'
gmBoxfill.xticlabels2 = '*'
gmBoxfill.yaxisconvert = 'linear'
gmBoxfill.ymtics1 = ''
gmBoxfill.ymtics2 = ''
gmBoxfill.yticlabels1 = '*'
gmBoxfill.yticlabels2 = '*'
kwargs = {}
kwargs[ 'cdmsfile' ] = cdmsfile.id
args.append( gmBoxfill )
canvas.plot( *args, **kwargs )

[doutriaux1]

@cameronsmith1 I've seen something similar on another X11-based system, it seems to be X related, because this person was using the exact same version that all my other users. One way around (painful) is to type canvas.interact(), that allows the redrawing once the window is in front. @dlonie @aashish24 is there a way around this in VTK? Basically until you type "interact" the interactor is not started on some X11 configs it seems to prevent the backing store as described here.

[doutriaux1]

@dlonie I assigned you, but I'm not sure there's much we can do, it might be on the user side?

[aashish24]

this is indeed a GL issue. I believe there is a workaround. @dlonie do you remember?

[alliepiper]

The vtkRenderWindow redraws itself automatically as needed in every other project I've used it in. Not sure why/how the UVCDAT implementation breaks this.

I doubt it's a GL issue, since it works in other projects on my system that use the same GL. Not sure what's going on with it here.

[alliepiper]

Unassigning myself because this is a usage issue from the VCS/Python side of things.

[doutriaux1]

I think it's X related. As I mentioned all of our users use the same version (mounted from same disk) but for some it autoredraws, for some it doesn't unless in interact mode. Could it be KDE vs Gnome?

[durack1]

@cameronsmith1 we had a bunch of problems which I was certain was X11 related a while back #1134), it turned out that it was a problematic X11 config on the client, for two independent user setups.. Once we cleaned up our setups independently, everything worked fine..

Do glxgears and glxinfo run smoothly on your setup?

[cameronsmith1]

@durack1, thanks for the suggestion. glxgears does and doesn't work. Specifically, if I run a single instance then it appears to work fine (the 3 gears turn, and I can cover and unvcover it with normal windows). However, if I run a second instance of glxgears, then the first one stops turning, and when I cover it with the second glxgears window the first one blanks and doesn't refresh -- until I kill the second glxgears.

The glxinfo produces a bunch of output. The one thing that looks to me like it might be helpful is "libGL error: failed to load driver: swrast".

[durack1]

@cameronsmith1 from memory this is an ongoing issue, I do recall speaking to @doutriaux1 about this a while back... Not sure if there is a solution, possibly upgrading/updating/checking your X11 client and making sure that you have all the pieces installed and up-to-date..

[durack1]

@dlonie here's a case where there's a window over the active canvas, and voila black in the region where the window overlays..

Should a separate issue be opened here, or is this related to the above?

Do you require any additional info about the configuration (X11, clients etc)..

[alliepiper]

I'm by no means an X expert, but from what I understand, a backing store is not the best solution here. They cause issues with modern compositing windowing systems, and most X implementations disable the backing store by default on their servers. Worse, the servers are free to ignore the backing store flags and override the behavior as they see fit. Thus, we'd need to make users update their X server configuration, and there's still no guarantee that it would work, or worse, it may cause rendering artifacts and tearing on certain platforms.

The 'proper' solution is to handle the X Expose window events to trigger a repaint. This isn't dependent on X server settings and doesn't have the compatibility issues that the backing store does. This is handled in VTK by the event loop implemented in vtkRenderWindowInteractor. When the window is shown and needs to be updated, vtkRenderWindowInteractor::Start should be called to start the event loop to handle the X events and update as the window gets re-exposed.

Looking at similar projects, this is how matplotlib works. You set up the visualization via calls to plot, etc, and then call show(), which creates the window and displays the visualization. Notice that this blocks the python interpreter until the window is closed -- you cannot interact with the python shell while the window is shown. That's the trade-off -- this method cannot be used from an interactive script and update re-exposed windows while the python prompt is shown, so showing the window becomes a blocking action.

At the moment, VTK does not have an option to enable to backing store on its render windows. Given the age and widespread use of VTK, that fact that this feature has not been added suggests to me that it's really not the ideal path forward. However, I can add an option to the X implementation of vtkRenderWindow to enable this, but that still won't address the problem on non-X platforms and it definitely won't work 'out of the box' for most users.

Let me know how you guys want to proceed.

[alliepiper]

Also, related to @durack1's image, I've seen this happen with some OpenGL drivers even with the interactor running (it happens to me with e.g. the latest nVidia linux drivers). My guess is that they're being super aggressive with the pixel-ownership test in the GL pipeline and discarding fragments that would be obscured by another window, saving a bit of time on blending operations and such. Which works fine, until you want to save out the window as an image, and glReadPixels returns a ton of black pixels because those pixels never received any fragments.... ;)

I deal with this by using a version of mesa (or some other opengl driver) when I need to get pixel data from obscured windows. This is hardly a good solution IMO, but since it's a driver issue in that case, there's not much else we can do about it.

[doutriaux1]

@dlonie I know it works when you starts the interactor, BUT once you started the interactor, your script pauses until the user press Q. There are two issues I have seen in "real life" happening here:
1- User starts a script windows appear in background (why I don't know), and moves to interactive mode so he/she can query variables. Well they can't see the image unless they type interact() and if they do then they can't use the prompt to query variables values etc...
2- User generated many pngs in foreground (because of #1397) now any window on top of it will lead to black squares.

Again as you said I'm not sure there's a way to fix this from within the application. I'm not too wrroied about (2) since really it should be ran in bg=1 mode. But 1 is a very frequent case. Also users have no idea they should type canvas.interact() but that's another story (I used to have a warning in there, but it was decided to take it off...)

[doutriaux1]

@dlonie in short I think you answered my initial question, there's not much we can do except asking the user to upgrade their X server...

[alliepiper]

Upgrading the X server won't change anything. Did you mean reconfigure (to enable backing store support)? That still wouldn't make a difference since the vtkXOpenGLRenderWindow implementation uses the default NotUseful backing store flag.

Just want to make sure we're on the same page here :)

[jypeter]

This thread reminds me a bit of what I have described in #1044 .

I also thought that maybe I could solve my problem by finding a way to enable the backing store, or maybe find an environment variable that I could set and solve everything by magic. My current kind-of unsatisfying workaround is to use the xfce window manager, that gives slightly better resultst than cinnamon

[durack1]

@jypeter here is my hacky work around to get things plotting and saving correctly to a png file:

# [user@ocean ~]$ xeyes ; # Should display in foreground
# [user@ocean ~]$ Xvfb :2 -screen 0 1600x1200x16
# [user@ocean ~]$ bg
# [user@ocean ~]$ setenv DISPLAY :2
# [user@ocean ~]$ xeyes ; # Should display in background
happy VCS plotting in "background" mode..

[doutriaux1]

@cameronsmith1 did you try to add bg=True to your plot command?

[doutriaux1]

@aashish24 let's talk about this. @dlonie suggestion might be worth considering if it doesn't take him too long.

[aashish24]

@doutriaux1 sure but may we should discuss all the options first. I am not sure relying on X is a good strategy. Are we targetting Supercomputers or desktops?

[jypeter]

@aashish24 Cases where I would like the graphics of UV-CDAT to work correctly (more on this at the end):

• UV-CDAT running on a local Linux machine (or Mac, I guess), that's the easiest
• UV-CDAT running in a Linux VirtualBox guest on a local Windows machine
• remote UV-CDAT (running on a Linux desktop, on a Linux server in the lab, or a remote supercomputer), with the output displayed:
  • on a Linux machine (or Mac), or Linux VirtualBox guest on a local Windows machine
  • on a Windows machine running some kind of X server (Xming, cygwin/X)

Do not forget that the local machine may not have a high end graphics card! Most of the desktop/laptops I buy here only have the default Intel HDxxxx graphics card

By work correctly I mean:

• as smoothly as matplotlib/basemap
• or if I use x.plot(v) and then x.interact(), I want to be able to easily (i.e. no lag) click on the vcs canvas and get the coordinates and value of the data where I clicked.
• or if I start uvcdat, I want to be able to resize the whole uvcdat window, and navigate in the pull-down menus at the top with no refresh lag
• and of course windows and parts of windows are refreshed correctly
• I can have several open canvases