getSymbols.yahoo() always throws "Error in new.session()" #358

abernal30 · 2022-04-28T19:45:07Z

Description

When applying the function getSymbols(), it errors and does not return a result.

Minimal, reproducible example

getSymbols("AAPL")
## Error in new.session() : Could not establish session after 5 attempts.

The text was updated successfully, but these errors were encountered:

pverspeelt · 2022-04-29T09:29:14Z

@joshuaulrich,

The issue is located in the function .getHandle and specifically in the local function new.session. All the 5 calls to the yahoo website return with no cookies, hence the function fails.

Looking over the code. I don't see much point in the .gethandle function as the yahoo query is also being built in the .yahooURL function. The only thing that .gethandle gives to .yahooURL is the crumb request to the query.srv. and cb has the value: "" for either query1 or query2. So this part is not needed to get the data.

Removing the .gethandle and removing the crumb part from building the yahoo url in .yahooURL returns the data without any issues.

mpadge · 2022-04-29T10:33:47Z

seeing same issue here

alihru · 2022-04-29T11:36:56Z

one also needs to get rid of the if statement inside the .getHandle function which stops the process if the number of cookies is zero

if (NROW(curl::handle_cookies(h)) ==0
stop(„Could not establish session after 5 attempts.“)

p.s. thank you for the quick fix! Really appreciate it

joshuaulrich · 2022-04-29T14:20:58Z

@pverspeelt and @alihru thanks for taking a look! I have a patch and am running CRAN and reverse-dependency checks now.

msperlin · 2022-04-29T14:25:40Z

Thanks for helping out @joshuaulrich . Im the author of BatchGetSymbols, which uses quantmod::getSymbols(), and got lots of emails this morning about the issue!

ThomasTranGithub · 2022-04-29T15:00:54Z

Seeing the same issue here.

datawookie · 2022-04-29T15:22:50Z

@joshuaulrich just confirming that I have tested the fix on the 358-getsymbols-new.session branch and it works for me. 👍 Thank you for responding so quickly! 🥇

asaeb1480 · 2022-04-29T15:36:25Z

what's the solution? These comments are not working!

datawookie · 2022-04-29T15:41:07Z

@asaeb1480 please try installing from the 358-getsymbols-new.session branch. It's working for me again.

asaeb1480 · 2022-04-29T15:42:21Z

What's the exact code for installation of 358-getsymbols-new.session plz?

datawookie · 2022-04-29T15:44:55Z

@asaeb1480 please look at https://cran.r-project.org/web/packages/githubinstall/vignettes/githubinstall.html.

joshuaulrich · 2022-04-29T15:53:14Z

remotes::install_github("joshuaulrich/[email protected]")

m1270 · 2022-04-29T16:02:15Z

Thank you @joshuaulrich

I can't install it due to this error:

Error: package ‘xts’ 0.12.1 was found, but > 0.12.1 is required by ‘quantmod’

The latest version of xts package is 0.12.1. I am running ubuntu 20.04 and R version of 4.1.2

joshuaulrich · 2022-04-29T16:03:33Z

@m1270 ah, that's because I have a change in development to move the S4 xts/zoo class to xts... I need to revert that. Thanks for the catch!

fededemo · 2022-04-29T16:10:40Z

Thanks joshua. getSymbols working now.
@m1270 working for me with an remotes::install_github("joshuaulrich/xts", force = TRUE)

mik3hall · 2022-04-29T16:21:52Z

That installs but doesn't appear to fix for me.

TommyTorstP · 2022-04-29T16:23:16Z

I have the same issue as @mik3hall here! I tried to find it in the given directory on the computer, but the file still seems to be deleted once install is complete

asaeb1480 · 2022-04-29T16:26:25Z

Thanks, Joshuaulrich. It's working for me.

mik3hall · 2022-04-29T16:30:15Z

@TommyTorstP it is in the user library path that it indicated in the install but I must not default search that. Looking at how I can include that now. I assume it will eventually cran install to a normal system path.

pawatkins · 2022-04-29T16:30:21Z

Hello,

I'm very sorry but I'm an idiot when it comes to this stuff. I'm currently enrolled in a master's class but I've only been coding since January 2022.

I have a project due tonight at midnight and it uses quantmod.

How can I fix this error? I'm in Rstudio

Thanks in advance. I know this has to be work.

joshuaulrich · 2022-04-29T16:33:07Z

@pawatkins remotes::install_github("joshuaulrich/[email protected]")

Baroni77 · 2022-04-29T16:39:04Z

after a bit of experimenting, it is working for me too. Thanks @joshuaulrich!

m1270 · 2022-04-29T16:44:44Z

Thank you @joshuaulrich
I confirm it is working for me as well.

MiraiLi00 · 2022-04-29T16:50:27Z

Hello, @joshuaulrich, thank you for helping with this issue, but I was trying to run the code you provided, but it is still not working for me and it gives me this error:

ERROR: package installation failed
Error: Failed to install 'quantmod' from GitHub:
System command 'Rcmd.exe' failed, exit status: 1, stdout & stderr were printed

(sorry I am new to this)

Do you know what could be a problem?

mik3hall · 2022-04-29T17:02:09Z

@TommyTorstP Still not working for me.

Installed to...
installing to /Users/mjh/Library/R/x86_64/4.1/library/00LOCK-xts/00new/xts/libs

portfolio <- c("SPY","ORCL","FCX","X","MRO","ZIM","USO")
from.dat <- as.Date("01/01/22", format="%m/%d/%y")
getSymbols(portfolio,from=from.dat,auto.assign=T)
‘getSymbols’ currently uses auto.assign=TRUE by default, but will
use auto.assign=FALSE in 0.5-0. You will still be able to use
‘loadSymbols’ to automatically load data. getOption("getSymbols.env")
and getOption("getSymbols.auto.assign") will still be checked for
alternate defaults.

This message is shown once per session and may be disabled by setting
options("getSymbols.warning4.0"=FALSE). See ?getSymbols for details.

Error in new.session() : Could not establish session after 5 attempts.

.libPaths()
[1] "/Users/mjh/Library/R/x86_64/4.1/library" "/Library/Frameworks/R.framework/Versions/4.1/Resources/library"

joshuaulrich · 2022-04-29T22:02:41Z

Hi @zlfang00. That's very odd and it's not clear why remotes::install_github() thinks the packages it just installed are not available. I could be missing something, or it may be a bug in the function. Either way, I'm afraid I can't help.

joshuaulrich · 2022-04-29T22:04:22Z

'package ‘quantmod’ is in use and will not be installed

@yw6010, that means you have an R session that already has quantmod loaded, so you can't change anything in the directory the package is installed in. You need to close all your R sessions on that computer, install quantmod, and then you can load and use quantmod.

ethanbsmith · 2022-04-29T22:12:19Z

rstudio can load and keep links to packages active for intellisense parsing and such, even after a Session -> Restart R. best to shut down everything and do installs from a clean vanilla r session.

yw6010 · 2022-04-29T22:14:48Z

'package ‘quantmod’ is in use and will not be installed

@yw6010, that means you have an R session that already has quantmod loaded, so you can't change anything in the directory the package is installed in. You need to close all your R sessions on that computer, install quantmod, and then you can load and use quantmod.

it works now!!! very appreciate. time to resume my project haha

rhamo · 2022-04-29T23:15:59Z

Thank you for the quick fix @joshuaulrich
Once installing "remotes::install_github("joshuaulrich/[email protected]")"
this worked for me once when requesting a small group of tickers, however all new attempts since I now constantly receive "Not Cached - Error in download.." errors.

Example:

I have tried restarting R but now I cannot even receive one successful return. Each ticker states "Error in download" once again. Any ideas?

Thank you

edwinhung · 2022-04-29T23:22:16Z

Hi, Thank you very much for the fix and comments here. getSymols.yahoo() now works for me, but I have a different problem. When I run getSymbols.yahoo() successfully in a loop for more than about 300-400 tickers, I started to get "HTTP error 401" for all following downloads. There are some failed downloads of invalid tickers in between though. Does anyone know what the issue is? Maybe I have the same problem with @rhamo.

Here is an example of the subsequent download:
> getSymbols.yahoo("AAPL",from="2022-04-27",to="2022-04-29",auto.assign=FALSE)
Warning: AAPL download failed; trying again. Warning: Unable to import “AAPL”. AAPL download failed after two attempts. Error message: HTTP error 401. [1] "Error in open.connection(file, \"rt\") : HTTP error 401.\n" attr(,"class") [1] "try-error" attr(,"condition") <simpleError in open.connection(file, "rt"): HTTP error 401.>

Thank you!

zpio · 2022-04-29T23:39:06Z

> install.packages("quantmod")
> library(quantmod)
> remotes::install_github("joshuaulrich/xts", force = TRUE)

* installing *source* package 'xts' ...
ERROR: cannot remove earlier installation, is it in use?
* removing 'C:/Users/Tama/Documents/R/win-library/3.6/xts'
* restoring previous 'C:/Users/Tama/Documents/R/win-library/3.6/xts'
Error in file.copy(lp, dirname(pkgdir), recursive = TRUE, copy.date = TRUE) : 
  (converted from warning) problem copying C:\Users\Tama\Documents\R\win-library\3.6\00LOCK-xts\xts\libs\x64\xts.dll to C:\Users\Tama\Documents\R\win-library\3.6\xts\libs\x64\xts.dll: Permission denied
* removing 'C:/Users/Tama/Documents/R/win-library/3.6/xts'
* restoring previous 'C:/Users/Tama/Documents/R/win-library/3.6/xts'
Error in file.copy(lp, dirname(pkgdir), recursive = TRUE, copy.date = TRUE) : 
  (converted from warning) problem copying C:\Users\Tama\Documents\R\win-library\3.6\00LOCK-xts\xts\libs\x64\xts.dll to C:\Users\Tama\Documents\R\win-library\3.6\xts\libs\x64\xts.dll: Permission denied
Execution halted
Error: Failed to install 'xts' from GitHub:
  (converted from warning) installation of package ‘C:/Users/Tama/AppData/Local/Temp/RtmpAnQpkC/file449050346c2/xts_0.12.1.2.tar.gz’ had non-zero exit status

> remotes::install_github("joshuaulrich/[email protected]")

 installing *source* package 'quantmod' ...
   ** using staged installation
   ** R
   ** demo
   ** byte-compile and prepare package for lazy loading
   Error: package 'xts' required by 'quantmod' could not be found
   Execution halted
   ERROR: lazy loading failed for package 'quantmod'
-  removing 'C:/Users/Tama/AppData/Local/Temp/RtmpqubKOV/Rinst31ac26a1569f/quantmod'
         -----------------------------------
   ERROR: package installation failed
Error: Failed to install 'quantmod' from GitHub:
  System command 'Rcmd.exe' failed, exit status: 1, stdout + stderr (last 10 lines):
E> ** using staged installation
E> ** R
E> ** demo
E> ** byte-compile and prepare package for lazy loading
E> Error: package 'xts' required by 'quantmod' could not be found
E> Execution halted
E> ERROR: lazy loading failed for package 'quantmod'
E> * removing 'C:/Users/Tama/AppData/Local/Temp/RtmpqubKOV/Rinst31ac26a1569f/quantmod'
E>       -----------------------------------
E> ERROR: package installation failed

I don't know what else to do. Help

BathosEconTrade · 2022-04-29T23:53:12Z

get this as well when apply patch:
"Warning: package ‘quantmod’ is in use and will not be installed"
this after I removed quantmod and Quandl and then new session then new install.
get this:
Error in new.session() : Could not establish session after 5 attempts.Error in new.session() : Could not establish session after 5 attempts.
Error in new.session() : Could not establish session after 5 attempts.
I am so grateful for quantmod and really dependent upon it

rpfreitasxyz · 2022-04-30T00:07:32Z

Hi, Thank you very much for the fix and comments here. getSymols.yahoo() now works for me, but I have a different problem. When I run getSymbols.yahoo() successfully in a loop for more than about 300-400 tickers, I started to get "HTTP error 401" for all following downloads. There are some failed downloads of invalid tickers in between though. Does anyone know what the issue is? Maybe I have the same problem with @rhamo.

Here is an example of the subsequent download: > getSymbols.yahoo("AAPL",from="2022-04-27",to="2022-04-29",auto.assign=FALSE) Warning: AAPL download failed; trying again. Warning: Unable to import “AAPL”. AAPL download failed after two attempts. Error message: HTTP error 401. [1] "Error in open.connection(file, \"rt\") : HTTP error 401.\n" attr(,"class") [1] "try-error" attr(,"condition") <simpleError in open.connection(file, "rt"): HTTP error 401.>

Thank you!

I've been having the same issue!

zlfang00 · 2022-04-30T00:09:46Z

Hi, @joshuaulrich, One thing I noticed is zoo is already installed in the public R library, while xts and TTR are not. My guess is this version quantmod only searches for its dependencies in the public library path?
Could you please confirm of that? Thanks very much!

Hi @zlfang00. That's very odd and it's not clear why remotes::install_github() thinks the packages it just installed are not available. I could be missing something, or it may be a bug in the function. Either way, I'm afraid I can't help.

BathosEconTrade · 2022-04-30T00:14:34Z

ok remote is working
then use getSymbols.yahoo("AAPL)
and still get:
Error in new.session() : Could not establish session after 5 attempts.
rpfreitasxyz you are doing better than I at least working but this this other problem.
any insights would be great.
all my models are based on getSymbols and yahoo, and am sure everyone notices is a very interesting time now.
many thanks

joshuaulrich · 2022-04-30T00:43:45Z

Patched quantmod on CRAN now

The source package for quantmod 0.4.20 has been released to CRAN! It may take another day or two for CRAN to build the Windows and macOS binaries, so users on those systems can install from source or wait for the binaries to be built.

Yahoo Finance alternative

I also want to mention an alternative to Yahoo Finance data. I've found the end of day data from tiingo to be higher quality and the endpoint is more stable than Yahoo Finance. You need to sign up to get a free API key, and that gives individuals free access with 500 requests/hour and 20,000 requests/day, up to 5GB of data/month.

Disclaimer: While I don't receive any referral compensation for pointing individuals to the Free Plan or Power Plan, I may receive compensation (I honestly don't remember how much) from them for new Commercial Plan customers who learned of tiingo through me. But you have to mention me when you join, I don't have an affiliate link or anything like that.

joshuaulrich · 2022-04-30T00:59:48Z

@rhamo, @edwinhung, and @rpfreitasxyz A HTTP 401 response means the request is "unauthorized". That can happen for various reasons, but my guess is that you're hitting rate limits imposed by the Yahoo Finance servers. getSymbols() adds pauses when you request a lot of symbols at once to try and avoid hitting these rate limits. @msperlin (author of BatchGetSymbols) may have a better idea.

Let's move this conversation to #360.

@zpio you shouldn't need to install xts from GitHub. You only need the version on CRAN. Can you try to install quantmod via remotes::install_github("joshuaulrich/quantmod") and see if that works?

@zlfang00 The new version of quantmod uses zoo, xts, and TTR just like prior versions. Nothing has changed in that regard, so I'm stumped about what's causing your issue.

Error in new.session() : Could not establish session after 5 attempts.

@BathosEconTrade That looks like version 0.4.18 is being loaded instead of 0.4.19 or 0.4.20. Check packageVersion("quantmod") to see which version you're loading. Double check your library paths in case you are using more than one library...

BathosEconTrade · 2022-04-30T11:47:52Z

Many thanks all! Great help. Must have version 4.2.

rhamo · 2022-05-01T02:30:49Z

Thank you for the quick fix @joshuaulrich Once installing "remotes::install_github("joshuaulrich/[email protected]")" this worked for me once when requesting a small group of tickers, however all new attempts since I now constantly receive "Not Cached - Error in download.." errors.

Example:

EMN | yahoo (161|506) | Found cache file - Got 100% of valid prices | Feels good! EMR | yahoo (162|506) | Found cache file - Got 100% of valid prices | Got it! ENPH | yahoo (163|506) | Found cache file - Got 100% of valid prices | Boa! EOG | yahoo (164|506) | Not Cached - Error in download.. EPAM | yahoo (165|506) | Not Cached - Error in download.. EQIX | yahoo (166|506) | Not Cached - Error in download.. EQR | yahoo (167|506) | Not Cached - Error in download.. ES | yahoo (168|506) | Not Cached - Error in download..

I have tried restarting R but now I cannot even receive one successful return. Each ticker states "Error in download" once again. Any ideas?

Thank you

Have tried with new quantmod 0.4.20 update but still no success. Every ticker states "Not Cached - Error in download" error as above.

Any ideas on how to solve this issue?
Thanks again

samiranvekar · 2022-05-01T02:44:01Z

Thank you@joshuaulrich. The issue is resolved

msperlin · 2022-05-01T11:56:42Z

It seems that yahoo finance is now restricting access to the data. Once you download too much data, you get the "HTTP 401" error. This is controled by Yahoo and there is nothing myself or @joshuaulrich can do about it.

As with any webscrapping script, you can try to overcome the restriction with:

changing your ip one you reach the limit (use VPN)
use a caching system to save data locally and download data over the course of days. Usually the restricted ips are released in 24 hours. A caching system is already available in yfR/BatchGetSymbols).

Overall, if the limits are too restrictive, it might be best to find other source of price data.

NilssonHedge · 2022-05-01T12:03:25Z

Is there any indication of what the Yahoo Rate limit is?

I can also confirm that the VPN fix is working, thanks for putting some serious effort into this. :)

msperlin · 2022-05-01T12:08:11Z

@NilssonHedge Im looking into it right now.. It doenst seem too restrictive. After a 401 error, I can download again in a few minutes.

zlfang00 · 2022-05-01T18:37:05Z

are you sure it is the yahoo restriction? I tried Python package yfinance to grab the same size of date, no issues.

joshuaulrich · 2022-05-02T00:47:00Z

are you sure it is the yahoo restriction? I tried Python package yfinance to grab the same size of date, no issues.

The yfinance python library uses a different endpoint that returns JSON instead of a CSV. I'll look into adding something to getSymbols() that tries the other endpoint if one of them fails.

rhamo · 2022-05-02T01:14:05Z

Changing ip has worked wonders. Once again BatchGetSymbols now retrieves data without errors.

Thanks so much for all your help @msperlin and @joshuaulrich

mif2000 · 2022-05-04T23:05:10Z

Hi, Really appreciate your dedication. I'm a new user to R (v 3.5.3) through Bert for Excel. I finally had everything working to call R functionality and getSymbols (with a single ticker) through an Excel VBA macro and then it abruptly stopped as noted in prior comments. It appears that the problem with getSymbols has been resolved and that I can now download quantmod v 4.20 zip file for Windows (10), which I did from the quantmod.zip link on the CRAN page dtd 2022-04-29, copying the unzipped quantmod directory to the BERT2/lib folder. The command "library (quantmod)" ouputs an error: "package or namespace load failed for 'quantmod' in rbind(info, getNamespaceInfo(env,"S3methods")). packageVersion("quantmod") outputs '0.4.20'. I must be doing something wrong! What is the proper way to install the new version of quantmod?

Thanks for your help,
mif2000

See: joshuaulrich/quantmod#358

joshuaulrich mentioned this issue Apr 28, 2022

Are Yahoo and FRED resources down? Is it temporary? #357

Closed

This was referenced Apr 29, 2022

ISSUE on quantmod::getSymbols() #359

Closed

Intermittent timeout issues with new getSymbols.yahoo as well as getSplits and getDividends #166

Closed

msperlin mentioned this issue Apr 29, 2022

BatchGetSymbols: Error in download msperlin/BatchGetSymbols#27

Closed

joshuaulrich closed this as completed in 0a90c3a Apr 30, 2022

rpfreitasxyz mentioned this issue Apr 30, 2022

HTTP Error 401 when called on large amount of tickers #360

Closed

Superlluz mentioned this issue Apr 30, 2022

Error message: Could not get cookie adrianmichel/yloader#16

Closed

joshuaulrich changed the title ~~Error in new.session()~~ getSymbols.yahoo() always throws "Error in new.session()" May 2, 2022

joshuaulrich mentioned this issue May 9, 2022

Add ability for getSymbols.yahoo() to import from JSON endpoint #362

Closed

jujuwong21 added a commit to jujuwong21/Stock-Visualizer-Shiny that referenced this issue Jan 22, 2023

Updated history b/c of broken function

d257a4f

See: joshuaulrich/quantmod#358

joshuaulrich added this to the Release 0.4.20 milestone Mar 27, 2023

getSymbols.yahoo() always throws "Error in new.session()" #358

getSymbols.yahoo() always throws "Error in new.session()" #358

Comments

abernal30 commented Apr 28, 2022 • edited by joshuaulrich Loading

Description

Minimal, reproducible example

pverspeelt commented Apr 29, 2022

mpadge commented Apr 29, 2022

alihru commented Apr 29, 2022

joshuaulrich commented Apr 29, 2022

msperlin commented Apr 29, 2022

ThomasTranGithub commented Apr 29, 2022

datawookie commented Apr 29, 2022

asaeb1480 commented Apr 29, 2022

datawookie commented Apr 29, 2022

asaeb1480 commented Apr 29, 2022

datawookie commented Apr 29, 2022

joshuaulrich commented Apr 29, 2022

m1270 commented Apr 29, 2022

joshuaulrich commented Apr 29, 2022

fededemo commented Apr 29, 2022

mik3hall commented Apr 29, 2022

TommyTorstP commented Apr 29, 2022

asaeb1480 commented Apr 29, 2022

mik3hall commented Apr 29, 2022

pawatkins commented Apr 29, 2022

joshuaulrich commented Apr 29, 2022

Baroni77 commented Apr 29, 2022

m1270 commented Apr 29, 2022

MiraiLi00 commented Apr 29, 2022

mik3hall commented Apr 29, 2022

joshuaulrich commented Apr 29, 2022

joshuaulrich commented Apr 29, 2022 • edited Loading

ethanbsmith commented Apr 29, 2022

yw6010 commented Apr 29, 2022 • edited by joshuaulrich Loading

rhamo commented Apr 29, 2022 • edited Loading

edwinhung commented Apr 29, 2022 • edited Loading

zpio commented Apr 29, 2022 • edited by joshuaulrich Loading

BathosEconTrade commented Apr 29, 2022

rpfreitasxyz commented Apr 30, 2022

zlfang00 commented Apr 30, 2022

BathosEconTrade commented Apr 30, 2022

joshuaulrich commented Apr 30, 2022 • edited Loading

Patched quantmod on CRAN now

Sponsors

Yahoo Finance alternative

joshuaulrich commented Apr 30, 2022 • edited Loading

BathosEconTrade commented Apr 30, 2022

rhamo commented May 1, 2022

samiranvekar commented May 1, 2022

msperlin commented May 1, 2022

NilssonHedge commented May 1, 2022 • edited Loading

msperlin commented May 1, 2022

zlfang00 commented May 1, 2022

joshuaulrich commented May 2, 2022

rhamo commented May 2, 2022

mif2000 commented May 4, 2022

abernal30 commented Apr 28, 2022 •

edited by joshuaulrich

Loading

joshuaulrich commented Apr 29, 2022 •

edited

Loading

yw6010 commented Apr 29, 2022 •

edited by joshuaulrich

Loading

rhamo commented Apr 29, 2022 •

edited

Loading

edwinhung commented Apr 29, 2022 •

edited

Loading

zpio commented Apr 29, 2022 •

edited by joshuaulrich

Loading

joshuaulrich commented Apr 30, 2022 •

edited

Loading

joshuaulrich commented Apr 30, 2022 •

edited

Loading

NilssonHedge commented May 1, 2022 •

edited

Loading