use current ZODB

[jimfulton]

bin/alltests passes for me.

I'm not set up to run the selenium tests, but am told that they're be tun automatically on a PR.

[jimfulton]

Please tell me you don't need a contributor's agreement to change some version pins. :)

[jensens]

I don't think we need a CLA for some version pins. It is an auto-generated message, part of our standard checks.

Thanks for the contribution. It is a big step forward for Plones effort to modernize its code base which includes to use most recent releases of its dependencies!

[tisto]

@jimfulton jenkins job is green!!! Thank you @jimfulton!

@plone/framework-team, @esteele do we need to discuss this first or can we merge right away?

[jimfulton]

Woo hoo! :)

[thet]

It doesn't break anything, right? So we can merge it, IMO.

[thet]

@jimfulton for trivial changes lines this, we didn't need a CLA :D (me, hiding away)

[tisto]

Personally I'm ok with merging. Though, since we are already in beta I guess @esteele should briefly confirm that merging is ok.

[jimfulton]

Given that this doesn't affect Plone features, I wouldn't think it would be a beta violation.
[shrug]

[ebrehault]

I am ok for merging too

[gforcada]

I'm fine with merging as well, specially if tests pass 😄 , but I do have a few questions:

• do we want to keep testing on the old version as well?
• are there no backwards compatible change?
• is there any migration needed? A migration guide is in place if that's the case?
• could theoretically one run on ZODB 3.x, copy the database on another server that runs 4.x or 5.x and then move it back to 3.x? Should we warn about that?

[jimfulton]

On Fri, Mar 31, 2017 at 3:18 PM, Gil Forcada Codinachs < ***@***.***> wrote: I'm fine with merging as well, specially if tests pass 😄 , but I do have a few questions: - do we want to keep testing on the old version as well?

If it's not to hard and you feel paranoid.

- are there no backwards compatible change?

Not in terms of applications. I recently discovered ZRS doesn't work with ZODB 5. I plan to fix that soon.

- is there any migration needed? A migration guide is in place if that's the case?

No.

- could theoretically one run on ZODB 3.x, copy the database on another server that runs 4.x or 5.x and then move it back to 3.x? Should we warn about that? AFAIK, but someone could try it. :)

FWIW, we updated a long-lived Pyramid project from ZODB 3 to ZODB 5 without any problems. But we didn't try to go back. People using ZEO who want to avoid downtime should update their clients and then update their server, at which point they can start taking advantage of new ZEO features, most-notably SSH.

[jimfulton]

@jamadden are you aware of any RelStorage issues people might run into switching to ZODB 5? Schemas haven't changed IIRC.

[hvelarde]

I'm also +1 on merging this, after fixing the issues with ZRS; I suggest making some noise with a news release on plone.org explaining what this big achievement means to Plone users.

@tkimnguyen is my man!

[esteele]

@jimfulton: I'll let you get away without a CLA. This time...

I know we have some groups using ZRS heavily (@vangheem in particular), but if it looks like something we can get fixed in the near future, I'm +1 to merge this.

[ale-rt]

After cloning:

• https://github.com/plone/simple-plone-buildout/tree/zodb5

and running ./bin/buildout -c zodb5.cfg, buildout suggests me the following additional pins:

BTrees = 4.4.1
ZEO = 5.0.4
persistent = 4.2.2
zodbpickle = 0.6.0
trollius = 2.1

Also the zeoserver is not starting:

Traceback (most recent call last):
  File "./bin/zeoserver", line 36, in <module>
    + sys.argv[1:]))
  File "/home/ale/.buildout/eggs/plone.recipe.zeoserver-1.3-py2.7.egg/plone/recipe/zeoserver/ctl.py", line 35, in main
    zeoctl.main(args)
  File "/home/ale/.buildout/eggs/ZEO-5.0.4-py2.7.egg/ZEO/zeoctl.py", line 28, in main
    zdaemon.zdctl.main(args, options)
  File "/home/ale/.buildout/eggs/zdaemon-2.0.7-py2.7.egg/zdaemon/zdctl.py", line 683, in main
    options.realize(args)
  File "/home/ale/.buildout/eggs/zdaemon-2.0.7-py2.7.egg/zdaemon/zdctl.py", line 106, in realize
    RunnerOptions.realize(self, *args, **kwds)
  File "/home/ale/.buildout/eggs/zdaemon-2.0.7-py2.7.egg/zdaemon/zdoptions.py", line 290, in realize
    self.load_schema()
  File "/home/ale/.buildout/eggs/zdaemon-2.0.7-py2.7.egg/zdaemon/zdoptions.py", line 338, in load_schema
    self.schema = ZConfig.loadSchema(self.schemafile)
  File "/home/ale/.buildout/eggs/ZConfig-2.9.3-py2.7.egg/ZConfig/loader.py", line 33, in loadSchema
    return SchemaLoader().loadURL(url)
  File "/home/ale/.buildout/eggs/ZConfig-2.9.3-py2.7.egg/ZConfig/loader.py", line 67, in loadURL
    return self.loadResource(r)
  File "/home/ale/.buildout/eggs/ZConfig-2.9.3-py2.7.egg/ZConfig/loader.py", line 195, in loadResource
    schema = ZConfig.schema.parseResource(resource, self)
  File "/home/ale/.buildout/eggs/ZConfig-2.9.3-py2.7.egg/ZConfig/schema.py", line 27, in parseResource
    xml.sax.parse(resource.file, parser)
  File "/home/ale/Code/venvs/buildout.python/parts/opt/lib/python2.7/xml/sax/__init__.py", line 33, in parse
    parser.parse(source)
  File "/home/ale/Code/venvs/buildout.python/parts/opt/lib/python2.7/xml/sax/expatreader.py", line 110, in parse
    xmlreader.IncrementalParser.parse(self, source)
  File "/home/ale/Code/venvs/buildout.python/parts/opt/lib/python2.7/xml/sax/xmlreader.py", line 123, in parse
    self.feed(buffer)
  File "/home/ale/Code/venvs/buildout.python/parts/opt/lib/python2.7/xml/sax/expatreader.py", line 213, in feed
    self._parser.Parse(data, isFinal)
  File "/home/ale/Code/venvs/buildout.python/parts/opt/lib/python2.7/xml/sax/expatreader.py", line 317, in start_element
    self._cont_handler.startElement(name, AttributesImpl(attrs))
  File "/home/ale/.buildout/eggs/ZConfig-2.9.3-py2.7.egg/ZConfig/schema.py", line 103, in startElement
    getattr(self, "start_" + name)(attrs)
  File "/home/ale/.buildout/eggs/ZConfig-2.9.3-py2.7.egg/ZConfig/schema.py", line 354, in start_section
    sectiontype = self.get_sectiontype(attrs)
  File "/home/ale/.buildout/eggs/ZConfig-2.9.3-py2.7.egg/ZConfig/schema.py", line 226, in get_sectiontype
    return self._schema.gettype(type)
  File "/home/ale/.buildout/eggs/ZConfig-2.9.3-py2.7.egg/ZConfig/info.py", line 334, in gettype
    raise ZConfig.SchemaError("unknown type name: " + `name`)
ZConfig.SchemaError: unknown type name: u'zeo'

This is the zeo.conf:

ale@emily:~/Code/plone/zodb5$ cat parts/zeoserver/etc/zeo.conf |egrep -v '^(\s*)$|^(\s*)#'
%define INSTANCE /home/ale/Code/plone/zodb5/parts/zeoserver
<zeo>
  address localhost:8100
  read-only false
  invalidation-queue-size 100
  pid-filename /home/ale/Code/plone/zodb5/var/zeoserver.pid
</zeo>
<filestorage 1>
  path /home/ale/Code/plone/zodb5/var/filestorage/Data.fs
  blob-dir /home/ale/Code/plone/zodb5/var/blobstorage
</filestorage>
<eventlog>
  level info
  <logfile>
      path /home/ale/Code/plone/zodb5/var/log/zeoserver.log
      format %(asctime)s %(message)s
    </logfile>
</eventlog>
<runner>
  program $INSTANCE/bin/runzeo
  socket-name /home/ale/Code/plone/zodb5/var/zeo.zdsock
  daemon true
  forever false
  backoff-limit 10
  exit-codes 0, 2
  directory $INSTANCE
  default-to-interactive true
  logfile /home/ale/Code/plone/zodb5/var/log/zeoserver.log
</runner>

Probably some work is needed for plone.recipe.zeoserver.

[jimfulton]

a) the config file looks like a concatenation of a ZEO config file and a zdaemon config file.
b) the traceback looks like zdaemon is trying to read it's config, in which case <zeo> wouldn't be valid.

Is this using some unholy ZConfig combination of zdaemon and ZEO config? If so, you have my sympathies. :)

The modern practice is to define a standard ZEO config, https://github.com/zopefoundation/ZEO#server-configuration, and then separately define a zdaemon config, as described here https://pypi.python.org/pypi/zdaemon, that has:

program PATHTORUNZEO -C PATHTOZEOCONFIG

[ale-rt]

The config file comes from this recipe:

https://github.com/plone/plone.recipe.zeoserver/blob/master/src/plone/recipe/zeoserver/__init__.py#L555

I think we may want to fiddle with it.

I also noticed we can upgrade the ZConfig version to 3.1.0.

[jimfulton]

Sigh. zeoctl. I hate these combined zdaemon scripts. I hadn't noticed that this was part of ZEO.

The zeoctl script is completely untested and, from your report, I guess changes made sometime broke it.

I'll have to see WTF. Grrrrrrr.

[jimfulton]

zopefoundation/ZEO#84

[jimfulton]

OK, well, I tried to fix zeoctl, but I don't know how to run it.

I built out ZODB 3.10.5 and tried running it with https://gist.github.com/jimfulton/ebe75021f869e4a7481ef30c345286fc:

bash-3.2$ bin/zeoctl -C zc.cfg fg
bin/runzeo
Error: no storages specified; use -f or -C
For help, use bin/runzeo -h

I suspect the fix is simple, but IDK how to run the dang thing.

I even tried building Plone (a trip) and it doesn't have one.

I tried googling for zeoctl, but couldn't find documentation.

I need some working example of zeoctl for some version of ZODB.

[tkimnguyen]

You want a plone.org news item for some version pin changes. Sure ;)

[jensens]

Given zeoctl is completely untested and it is not the "standard" way to start ZEO:
IMO we should better move to the standard way to configure and start ZEO instead of fighting with a broken script.

As far as I understand it, the recommended way is using either runzeo or zdaemon together with runzeo. From my point of view Supervisor with runzeo (or similar) would be an option too for production deployments. For Systemd based Linux systems runzeo could be controlled directly by it.

Without having had a look at the details: For BBB/convenience reasons the plone.recipe.zeoserver could generate a short script starting runzeo with zdaemon named zeoctl?

[jimfulton]

I just released ZEO 5.1.0, which might plausibly fix the zeoctl issue. IDK how to run zeoctl, o I can't say for sure.

[jimfulton]

On Mon, Apr 3, 2017 at 4:13 AM, Jens W. Klein ***@***.***> wrote: Given zeoctl is completely untested and it is not the "standard" way to start ZEO: IMO we should better move to the standard way to configure and start ZEO instead of fighting with a broken script.

Yup. Unfortunately, I think it was a "standard" way a long time ago.

As far as I understand it, the recommended way <https://pypi.python.org/pypi/ZEO/4.2.0b1#installing-software> is using either runzeo or zdaemon together with runzeo. From my point of view Supervisor with runzeo (or similar) would be an option too for production deployments. For Systemd based Linux systems runzeo could be controlled directly by it.

The right way to think about deploying ZEO is to decouple running ZEO from daemonizing it. - Write/generate a ZEO configuration, which is passed to runzeo using the -C option. - daemonize the runzeo process using a wrapper like zdaemon (better, IMO for automated deployments) or supervisord (maybe better for manual deployments, but what are those? ;-) ) or whatever else works (systemd? IDK). zeoctl should be considered walking (or perhaps not) dead. I want Plone to start using current ZODB, so I'd like to require fewer short-term changes to Plone. I just released ZEO 5.1. It has a change that might plausibly fix the zeoctl issue. Given that I don't know and can't find out how ti successfully run zeoctl even with older ZODB releases, I can't really test the change.

[ale-rt]

Thanks @jimfulton, ZEO = 5.1.0 did the trick!

See complete buildout here:

• https://github.com/plone/simple-plone-buildout/compare/master...plone:zodb5?expand=1

So I am fine for merging but I would say we should also add some more pins.
I will try to run ploneintranet test suite (which runs against a ZEO cluster and report back) .

Thanks again @jimfulton, your contribution (as you can see by the enthusiastic reaction of all these Plonistas) is really appreciated!

[jimfulton]

On Mon, Apr 3, 2017 at 4:45 PM, Alessandro Pisa ***@***.***> wrote: Thanks @jimfulton <https://github.com/jimfulton>, ZEO = 5.1.0 did the trick!

Yay! When y'all have some time later, please stop using zeoctl. It's still untested. :) See complete buildout here:

- https://github.com/plone/simple-plone-buildout/compare/ master...plone:zodb5?expand=1 So I am fine for merging but I would say we should also add some more pins. I will try to run ploneintranet test suite (which runs against a ZEO cluster and report back) . Thanks again @jimfulton <https://github.com/jimfulton>, your contribution (as you can see by the enthusiastic reaction of all these Plonistas) is really appreciated!

You're more than welcome. I'm glad that Plone users will soon be using the latest greatest ZODB.

[jimfulton]

Also, I'm putting finishing touches on a ZRS release that works with ZODB 5.

[jensens]

I Love it!

Things to do before merge:

• update version annotation comment for zodb3 at the bottom of versions.cfg
• include version pin for ZEO

[jensens]

I will do a manual local merge and add those two changes, GTD!

[jensens]

manual merge of this PR with fb67f9e together with a second commit e4d9d1c with above mentioned changes.

This was on our wishlist for along time. Thanks @jimfulton for making this possible!

[gforcada]

@jensens sorry, but although you want it for a long time that does not mean that merging it right now was a good idea. There is quite a lot of ZRS installations out there and until jim finishes the release and it can be tried in the wild I would have very much preferred to not have this merged.

Is only on 5.1 and we are still in beta, so nothing much to panic about, but for such big changes, giving the chance to get some early feedback, like @ale-rt did has been crucial to fix a few things already.

And anyway, as long as there is no release, the version pin bumps are only on git, so unless you extend buildout.coredev git it does not really matter if it's merged or not.

I would have rather announced it on community.plone.org with the set of new pins to test, to get some more testing before getting it merged.

[jensens]

Looks like the tests that were green here are red now, and I have no good idea why that is that way, so I'll revert.

[jensens]

@gforcada I am sure the ZRS issue will get fixed.

From our real world experience people do not test branches (excpet those readin the PRs). Poor enough, I know.

[sneridagh]

Totally agree on @gforcada opinion. We are in beta for 5.1, if that potentially breaks something (it actually does) or untested in the wild (that's why we have -pending releases) I would add it to later (and better tested) releases. Can we just add an experimental cfg for people to fiddle with it in the meanwhile?

…

On Wed, 5 Apr 2017 at 11:32, Gil Forcada Codinachs ***@***.***> wrote: @jensens <https://github.com/jensens> sorry, but although you want it for a long time that does not mean that merging it right now was a good idea. There is quite a lot of ZRS installations out there and until jim finishes the release and it can be tried in the wild I would have very much preferred to not have this merged. Is only on 5.1 and we are still in beta, so nothing much to panic about, but for such big changes, giving the chance to get some early feedback, like @ale-rt <https://github.com/ale-rt> did has been crucial to fix a few things already. And anyway, as long as there is no release, the version pin bumps are only on git, so unless you extend buildout.coredev git it does not really matter if it's merged or not. I would have rather announced it on community.plone.org with the set of new pins to test, to get some more testing before getting it merged. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#317 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAduD6w6rjAhxnHBtD9DGbOO81RimLTMks5rs1vxgaJpZM4Mv1rr> .

[jensens]

After all I would like to see this in 5.1. But only if it does not break anything. Otherwise we need to wait for 6.0 and that would be really sad.

[gforcada]

@jensens don't get me wrong, I do want to have that on 5.1 as well, but if we already know that there is something that does really not work (ZRS) it does not make sense to merge and wait for "it will get fixed".

Sure it will, hopefully, but on the meantime all and everyone that uses ZRS will not be able to check 5.1 as they could have been doing because of that. Let's follow a process please, not just "let me click the green button because I feel like it".

We could even argue that was supposed to be the responsibility of the @plone/release-team actually. Putting releases together is part of our (as I'm on that team) job, so bumping versions should be approved by us.

[jensens]

Well, increasing version pins of our dependencies was never done explicitly by the release team. I would wish there was that role. But in fact, a small group of contributors tracked the versions and did that job.

I agree ZRS is important. @jimfulton stated already "I recently discovered ZRS doesn't work with ZODB 5. I plan to fix that soon." So I have no doubt that this will turn into a show stopper. Also, ZRS is not an integral part of Plone core, nor is it part of/ pinned in Plone ecosystem.

[jensens]

OK, the ZRS problem just vanished. Thanks again @jimfulton
https://pypi.python.org/pypi/zc.zrs/3.0.0
https://groups.google.com/forum/#!topic/zodb/NPuZFGoC-6c

[gforcada]

@jensens as for a role to update version pins, IMHO anyone should /be encouraged to suggest version pin upgrades, but it should be the release team the one merging those... that's the role of a release team, get releases out and compile the KGS.

[ale-rt]

BTW I still had no luck to run ploneintranet test suite either:

Running ploneintranet.workspace.testing.PloneintranetworkspaceLayer:Functional tests:
  Tear down ploneintranet.search.solr.testing.PloneIntranetSearchSolrLayer:Robot in 0.000 seconds.
  Tear down plone.app.robotframework.testing.Plone:Robot in 0.000 seconds.
  Tear down plone.app.robotframework.testing.PloneRobotFixture in 0.003 seconds.
  Tear down plone.app.robotframework.testing.SimplePublicationLayer in 0.006 seconds.
  Tear down plone.app.robotframework.testing.MockMailHostLayer in 0.010 seconds.
  Tear down plone.app.robotframework.remote.RemoteLibraryBundle:RobotRemote in 0.007 seconds.
  Tear down ploneintranet.search.solr.testing.PloneIntranetSearchSolrTestContentLayer Traceback (most recent call last):
  File "bin/test", line 380, in <module>
  File "/var/lib/jenkins/.buildout/eggs/collective.xmltestreport-1.3.3-py2.7.egg/collective/xmltestreport/runner.py", line 60, in run
    failed = run_internal(defaults, args, script_parts=script_parts)
  File "/var/lib/jenkins/.buildout/eggs/collective.xmltestreport-1.3.3-py2.7.egg/collective/xmltestreport/runner.py", line 73, in run_internal
    runner.run()
  File "/var/lib/jenkins/.buildout/eggs/zope.testrunner-4.4.10-py2.7.egg/zope/testrunner/runner.py", line 171, in run
    self.run_tests()
  File "/var/lib/jenkins/.buildout/eggs/zope.testrunner-4.4.10-py2.7.egg/zope/testrunner/runner.py", line 253, in run_tests
    self.skipped, self.import_errors)
  File "/var/lib/jenkins/.buildout/eggs/zope.testrunner-4.4.10-py2.7.egg/zope/testrunner/runner.py", line 405, in run_layer
    tear_down_unneeded(options, needed, setup_layers)
  File "/var/lib/jenkins/.buildout/eggs/zope.testrunner-4.4.10-py2.7.egg/zope/testrunner/runner.py", line 696, in tear_down_unneeded
    l.tearDown()
  File "/var/lib/jenkins/.buildout/eggs/plone.app.testing-5.0.6-py2.7.egg/plone/app/testing/helpers.py", line 412, in tearDown
    self.tearDownZope(app)
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/var/lib/jenkins/.buildout/eggs/plone.testing-5.0.0-py2.7.egg/plone/testing/z2.py", line 317, in zopeApp
    connection.close()
  File "/var/lib/jenkins/.buildout/eggs/ZODB-5.2.0-py2.7.egg/ZODB/Connection.py", line 283, in close
    raise ConnectionStateError("Cannot close a connection joined to "
ConnectionStateError: Cannot close a connection joined to a transaction

I can fix this error adding a transaction.commit() before tearing down zope, but I do not know how clean is this solution.

[jimfulton]

I would to an abort in the tearDown before calling close.

[jensens]

@ale-rt I doubt a commit in a test is a good idea. As you noticed I added the recommended abort. @icemac already reviewed it and requested changes. It would be great if you can test plone.testing master with ploneintranet (the version w/o commits).

[ale-rt]

@jensens I will do that probably this evening

[jamadden]

@jamadden are you aware of any RelStorage issues people might run into switching to ZODB 5? Schemas haven't changed IIRC.

I'm not aware of any RelStorage-specific significant issues. There are no schema changes that require a migration. It's been a drop-in upgrade for us from 1.6.0bX all the way through to 2.1a1. That said, a few things of note:

• Python 2.7 or above is required (same with ZODB, though)
• The recommended MySQL client has changed and can be installed with the [mysql] extra
• Delayed polling (poll-interval) is not supported anymore (the option will be ignored)
• CFFI is required; binary wheels are distributed but if you can't install those you'll need a compiler
• Support for very old Oracle and PostgreSQL versions has been dropped
• The code has been restructured and internals are in completely different places (although the old structure wasn't publicly documented or recommended for use anyway)

(Apologies for the delay, I've had a bad case of the flu.)

[ale-rt]

@jensens in order for the build to run correctly I had to add another commit,
I added it in a separate branch, see https://github.com/plone/plone.testing/tree/abort-before-close-ale.
Do you want me to cherry-pick plone/plone.testing@ce51de4 in your PR?

[jensens]

@ale-rt thanks. Yes for cherry picking. It would be great if we can fix this in one PR.

[jensens]

did it, see plone/plone.testing#31