Fix optional internet tests. #13884

robertwb · 2012-12-29T23:03:34Z

Untested = broken.

sage -t --only_optional=internet devel/sage/sage/symbolic/integration/integral.py # 2 doctests failed
sage -t --only_optional=internet devel/sage/sage/combinat/words/paths.py # 1 doctests failed
sage -t --only_optional=internet devel/sage/sage/misc/preparser.py # 1 doctests failed
sage -t --only_optional=internet devel/sage/sage/interfaces/r.py # 1 doctests failed
sage -t --only_optional=internet devel/sage/sage/finance/stock.py # 11 doctests failed
sage -t --only_optional=internet devel/sage/sage/databases/sloane.py # 8 doctests failed

The files sage/databases/sloane.py and sage/combinat/words/paths.py are really covered by #10358.

The file sage/interfaces/r.py is a situation needing clarification at #7771.

This ticket is to fix the minor issues in the integral, preparer, and stock files.

Apply:

Component: doctest coverage

Author: Karl-Dieter Crisman

Branch: 90a4128

Reviewer: Ralf Stephan, Volker Braun

Issue created by migration from https://trac.sagemath.org/ticket/13884

The text was updated successfully, but these errors were encountered:

robertwb · 2012-12-30T00:29:14Z

comment:1

So this doesn't happen again, re-enable these via #13540.

kcrisman · 2013-01-03T17:49:40Z

comment:2

sage -t --only_optional=internet devel/sage/sage/interfaces/r.py

Just for clarification, this one is broken, but not because of the internet, and has been for a while. This was originally noticed (and fixed) by Minh while reviewing #7771 in this patch. It wasn't clear how to proceed, as you can see by reading the comments there.

kcrisman · 2013-01-03T17:54:09Z

comment:3

Haha! And the first one is not really a problem, just a logic issue.

sage -t --only_optional=internet devel/sage/sage/symbolic/integration/integral.py
    sage: f.integrate(algorithm="mathematica_free")       # optional -- requires internet
Expected:
    sqrt(pi)*sqrt(1/2)*fresnels(sqrt(2)*x/sqrt(pi)) + y^z*x
Got:
    doctest:674: DeprecationWarning: Variable of integration should be specified explicitly.
    See http://trac.sagemath.org/12438 for details.
    sqrt(pi)*sqrt(1/2)*fresnels(sqrt(2)*x/sqrt(pi)) + y^z*x
**********************************************************************
Expected:
    doctest:...: DeprecationWarning:
    Variable of integration should be specified explicitly.
    See http://trac.sagemath.org/12438 for details.
    1/2*x^2
Got:
    1/2*x^2

What happens is that when the test is done, the deprecation warning, which only occurs once, occurs with the internet test. This is a two-character fix. I'll try to see if I can fix any others along with it.

kcrisman · 2013-01-03T17:56:35Z

comment:4

Next: Considering that William at some point changed the file in this test

    sage: sage.misc.preparser.load('http://wstein.org/loadtest.py', globals())  # optional - internet
Expected:
    hi from the net
Got:
    hi from the net
    5

to

print "hi from the net"

print 2+3

I think we'll be okay just changing this one too.

kcrisman · 2013-01-03T18:05:41Z

comment:5

Just for reference, in general this file (sloane.py) is almost completely undocumented and #10358 is supposed to update with respect to this. So this is fully known.

kcrisman · 2013-01-03T18:10:11Z

comment:6

The finance errors seem mostly due to a change in how Google uses startdate, though at least one of them shouldn't have even worked the first time!


    sage: finance.Stock('vmw').google()[:5]   # optional -- internet
Expected:
    [
     28-Nov-07 80.57 88.49 80.57 87.69    7496000,
     29-Nov-07 90.91 93.20 89.50 90.85    5497600,
     30-Nov-07 95.39 95.60 89.85 91.37    4750200,
      3-Dec-07 89.87 96.00 88.70 94.97    4401100,
      4-Dec-07 92.26 97.10 92.05 95.08    2896600
    ]
Got:
    [
      5-Jan-12 81.76 82.84 81.52 82.31    2888520,
      6-Jan-12 82.96 84.33 82.47 83.31    1922781,
      9-Jan-12 83.45 85.17 82.94 84.95    1841373,
     10-Jan-12 86.01 86.80 85.10 85.30    1762911,
     11-Jan-12 85.42 88.20 85.30 87.92    2192664
    ]

At least this tells us when William added the finance stuff!

kcrisman · 2013-01-03T18:12:06Z

comment:7

And finally, the combinat error is due to the broken sloane_find issue. So this ticket should be quite fixable - hope to have a patch up soon.

kcrisman · 2013-01-03T18:42:11Z

comment:9

Google now apparently does not return data that is more than a certain number of years old - or at least the interface must have changed?


sage: finance.Stock('F').google('Jan+3,+1978', 'Jul+7,+2008')[:1]
[
 20-Aug-92 0.00 2.26 2.21 2.24   19202479
]
sage: finance.Stock('F').google('Jan+3,+1982', 'Jul+7,+2008')[:1]
[
 20-Aug-92 0.00 2.26 2.21 2.24   19202479
]

In fact, according to this blog post,

These APIs are now deprecated but have no scheduled shutdown date: ... Finance API ...

And although the data is still there if one uses e.g. a URL like this downloading the CSV only gives you so much.

What should we do, oh many Sage devels who are Google employees? ;-) I can fix some stuff but don't want to fix more of this if it's going to be gone eventually anyway.

kcrisman · 2013-01-03T19:16:22Z

comment:10

Okay, the following patch fixes everything not from #10358, #7771, and the change in the CSV files that the Google Finance API produces. (Note that Apple must have had a 4-1 stock split at some point as well.) In some cases I could circumvent the latter when the point wasn't testing historical data, but there are still two failures.


sage -t --only_optional=internet "devel/sage/sage/finance/stock.py"
**********************************************************************
    sage: finance.Stock('F').google('Jan+3,+1978', 'Jul+7,+2008')[:5] # optional -- internet
Expected:
    [
      3-Jan-78 0.00 1.93 1.89 1.89    1618200,
      4-Jan-78 0.00 1.89 1.87 1.88    2482700,
      5-Jan-78 0.00 1.89 1.84 1.84    2994900,
      6-Jan-78 0.00 1.84 1.82 1.83    3042500,
      9-Jan-78 0.00 1.81 1.79 1.81    3916400
    ]
Got:
    [
     20-Aug-92 0.00 2.26 2.21 2.24   19202479,
     21-Aug-92 0.00 2.27 2.19 2.20   18689616,
     24-Aug-92 0.00 2.17 2.10 2.10   38652825,
     25-Aug-92 0.00 2.19 2.11 2.18   31028049,
     26-Aug-92 0.00 2.21 2.18 2.20   22539411
    ]
**********************************************************************
    sage: sage.finance.stock.Stock("AAPL", 22144).google(startdate='Jan+1,+1990')[:5] #optional -- internet
Expected:
    [
      2-Jan-90 0.00 9.38 8.75 9.31    6542800,
      3-Jan-90 0.00 9.50 9.38 9.38    7428400,
      4-Jan-90 0.00 9.69 9.31 9.41    7911200,
      5-Jan-90 0.00 9.56 9.25 9.44    4404000,
      8-Jan-90 0.00 9.50 9.25 9.50    3627600
    ]
Got:
    [
     28-Feb-97 0.00 1.05 1.02 1.02   17411200,
      3-Mar-97 0.00 1.03 1.00 1.01   18636800,
      4-Mar-97 0.00 1.03 1.00 1.03   14745600,
      5-Mar-97 0.00 1.06 1.03 1.06   13737600,
      6-Mar-97 0.00 1.06 1.03 1.04   16612800
    ]
**********************************************************************

I don't know if these are fixable.

kcrisman · 2013-01-03T19:20:30Z

comment:11

Attachment: trac_13884.patch.gz

Patchbot, apply trac_13884.patch

I'd appreciate feedback; however, the easiest to me seems to be to review this and then open a new ticket to fix the Google finance interface csv download issue - if that is even possible.

kcrisman · 2013-01-03T19:20:30Z

Author: Karl-Dieter Crisman

kcrisman · 2013-06-14T15:30:20Z

comment:12

Okay, in the meantime the Google stuff has changed yet again (???), and we get the following bizarre error:

File "sage/symbolic/integration/integral.py", line 466, in sage.symbolic.integration.integral.integrate
Failed example:
    f.integrate(algorithm="mathematica_free")       # optional - internet
    NameError: name 'f' is not defined
**********************************************************************

This one baffles me; even doing f.integrate(x) with no algorithm or anything still fails. Doing this by hand usually works - I think there is some potential for timeouts. But it consistently fails with this in doctesting. So I'm giving up on this patch for now, unless someone has some other bright ideas.

fchapoton · 2013-09-01T15:40:59Z

comment:15

Attachment: trac_13884_addon1.patch.gz

I think this solves the problem in the integral file.

kcrisman · 2013-09-02T13:14:18Z

comment:16

The add-on patch is good. I'm not sure why that should make a difference, but it did.

I'm tempted to have us just add the preparser test to the integral one and then open a new ticket for stock.py, which clearly isn't functioning properly in any case (not just a problem of changed values any more). Frederic, do you want to create a patch like that?

vbraun · 2014-01-04T04:24:01Z

comment:17

A partial fix is better than no fix at all, IMHO.

robertwb · 2014-01-04T05:14:52Z

comment:18

I agree, no need to fix all of them before any of them go in.

fchapoton · 2014-01-04T10:26:38Z

Commit: 90a4128

fchapoton · 2014-01-04T10:26:38Z

Branch: u/chapoton/13884

fchapoton · 2014-01-04T10:26:38Z

New commits:

`90a4128`	`Trac #13884 - fix most broken internet optional doctests`

rwst · 2014-04-05T08:48:56Z

comment:21

What test command do you use? With --optional=internet I get:

sage -t src/sage/finance/stock.py  # 11 doctests failed
sage -t src/sage/symbolic/integration/external.py  # 1 doctest failed

vbraun · 2014-04-05T10:47:40Z

Reviewer: Ralf Stephan, Volker Braun

vbraun · 2014-04-05T10:47:40Z

comment:22

Sounds good to me; somebody interested in finance can take up the stock.py failures.

vbraun · 2014-04-06T20:49:06Z

Changed branch from u/chapoton/13884 to 90a4128

fchapoton · 2015-09-06T13:37:27Z

Changed commit from 90a4128 to none

robertwb added this to the sage-5.11 milestone Dec 29, 2012

robertwb added t: tests labels Dec 29, 2012

robertwb assigned sagetrac-mvngu Dec 29, 2012

This comment has been minimized.

Sign in to view

kcrisman added the s: needs review label Jan 3, 2013

kcrisman added s: needs work and removed s: needs review labels Jun 14, 2013

jdemeyer modified the milestones: sage-5.11, sage-5.12 Aug 13, 2013

This comment has been minimized.

Sign in to view

fchapoton added s: needs review and removed s: needs work labels Jan 4, 2014

sagetrac-vbraun-spam mannequin modified the milestones: sage-6.1, sage-6.2 Jan 30, 2014

vbraun added s: positive review and removed s: needs review labels Apr 5, 2014

vbraun removed the s: positive review label Apr 6, 2014

vbraun closed this as completed in 9c61128 Apr 6, 2014

This comment has been minimized.

Sign in to view

burcin mentioned this issue Jan 11, 2013

optional doctest failure -- broken finance doctest failures #6334

Closed

robertwb mentioned this issue Aug 30, 2016

Automatically detect and test optional dependencies #13540

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix optional internet tests. #13884

Fix optional internet tests. #13884

robertwb commented Dec 29, 2012

robertwb commented Dec 30, 2012

kcrisman commented Jan 3, 2013

kcrisman commented Jan 3, 2013

kcrisman commented Jan 3, 2013

kcrisman commented Jan 3, 2013

kcrisman commented Jan 3, 2013

kcrisman commented Jan 3, 2013

This comment has been minimized.

kcrisman commented Jan 3, 2013

kcrisman commented Jan 3, 2013

kcrisman commented Jan 3, 2013

This comment has been minimized.

kcrisman commented Jan 3, 2013

kcrisman commented Jun 14, 2013

This comment has been minimized.

This comment has been minimized.

fchapoton commented Sep 1, 2013

kcrisman commented Sep 2, 2013

vbraun commented Jan 4, 2014

robertwb commented Jan 4, 2014

fchapoton commented Jan 4, 2014

fchapoton commented Jan 4, 2014

fchapoton commented Jan 4, 2014

rwst commented Apr 5, 2014

vbraun commented Apr 5, 2014

vbraun commented Apr 5, 2014

vbraun commented Apr 6, 2014

This comment has been minimized.

fchapoton commented Sep 6, 2015

Fix optional internet tests. #13884

Fix optional internet tests. #13884

Comments

robertwb commented Dec 29, 2012

robertwb commented Dec 30, 2012

kcrisman commented Jan 3, 2013

kcrisman commented Jan 3, 2013

kcrisman commented Jan 3, 2013

kcrisman commented Jan 3, 2013

kcrisman commented Jan 3, 2013

kcrisman commented Jan 3, 2013

This comment has been minimized.

kcrisman commented Jan 3, 2013

kcrisman commented Jan 3, 2013

kcrisman commented Jan 3, 2013

This comment has been minimized.

kcrisman commented Jan 3, 2013

kcrisman commented Jun 14, 2013

This comment has been minimized.

This comment has been minimized.

fchapoton commented Sep 1, 2013

kcrisman commented Sep 2, 2013

vbraun commented Jan 4, 2014

robertwb commented Jan 4, 2014

fchapoton commented Jan 4, 2014

fchapoton commented Jan 4, 2014

fchapoton commented Jan 4, 2014

rwst commented Apr 5, 2014

vbraun commented Apr 5, 2014

vbraun commented Apr 5, 2014

vbraun commented Apr 6, 2014

This comment has been minimized.

fchapoton commented Sep 6, 2015