It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
Generally my fork should

Faster for updating the manifest, especially for full manifest updates (becaue it doesn't pause between every file for significant time) particularly if you have games with lots of individual files. This may be less true for update only / new only updates because the resume is much more paranoid because touching the games clear the flags but for games with lots of files / extras it should still be a net benefit.

A little slower to download the actual game files (due to preallocation and download resume support requiring MD5 checking of chunks and downloading the MD5 chunk file). But these should be a net benefit overall (defragmentation avoidance should be very useful on Windows non-SSD drives over time) and download resume should be a massive help on Linux which can have some very large file installers.

The first verification should be a little bit slower but later verifications should be much faster since they only verify things that have changed by default.


Significant deviations from this probably indicate a problem of some kind so I'd like to know details.
avatar
kohlrak: If you buy a game, it goes in the DB no problem. If you buy DLC, goodies, or whatever for a game that's already in the DB, since it doesn't get marked updated, and since it's for a game already in the database, it ignores it, thus you can end up with alot of stuff missing from the updates.
avatar
timppu: I presume this happens only if you keep the old manifest file (when running update)?

To me that wouldn't really seem like a bug in gogrepo, it is merely a "feature" in the GOG store (ie. the games for which you buy new DLC don't get marked as "new" (or "updated", for that matter) in your account.

At least the workaround is rather simple in gogrepo, I believe: delete the old manifest file and run gogrepo update, because then you are not relying on those "new" or "updated" flags, but you are guaranteed to receive any changed or added files. Maybe there was also already a separate gogrepo option to do a full update, even if you have an existing manifest file...
The problem with this is that it ends up taking a very, very long time. So i do this periodically, and since i'm deleting both databases to be safe, i'm also having to redo checksums, and on an external USB drive, this whole process actually ends up taking days. Oh well, i forgot about full updates, so thank you for reminding me. They take over an hour i think, but it's better than choosing between "days waiting" and "not getting everything".
avatar
timppu: Maybe there was also already a separate gogrepo option to do a full update, even if you have an existing manifest file...
There absolutely is. I do it once or twice a month.
Huh. I got a pointer error, leading to a segfault, and now attempting to resume download the GOG servers are 104 (reset by peer) every time. Maybe they don't like downloading terabytes at a time over a few days?

10:01:20 | request failed: HTTPSConnectionPool(host='content-system.gog.com', port=443): Max retries exceeded with url: /products/1430392753/users/46987611140699792/namespaces/website/download?path=%2Fsecure%2Fshadow_warrior_2013% 2Fwin%2Fdx9_version%2Fnew%2Fsetup_shadow_warrior_2.2.0.1-2.bin&access_token=REMOVEDTOKEN (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0xb4dc318c>, 'Connection to content-system.gog.com timed out. (connect timeout=30)')) (3 retries left) -- will retry in 5s...
If a download set goes a while, do I have to refresh the catalog with update again?
Post edited July 11, 2018 by mqstout
avatar
mqstout: Huh. I got a pointer error, leading to a segfault, and now attempting to resume download the GOG servers are 104 (reset by peer) every time. Maybe they don't like downloading terabytes at a time over a few days?

10:01:20 | request failed: HTTPSConnectionPool(host='content-system.gog.com', port=443): Max retries exceeded with url: /products/1430392753/users/46987611140699792/namespaces/website/download?path=%2Fsecure%2Fshadow_warrior_2013% 2Fwin%2Fdx9_version%2Fnew%2Fsetup_shadow_warrior_2.2.0.1-2.bin&access_token=REMOVEDTOKEN (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0xb4dc318c>, 'Connection to content-system.gog.com timed out. (connect timeout=30)')) (3 retries left) -- will retry in 5s...
avatar
mqstout: If a download set goes a while, do I have to refresh the catalog with update again?
I noticed around a week ago when switching to full updates with gogrepo, my cookies have been expiring.
Download kept failing until I did login AND update again.
Post edited July 11, 2018 by mqstout
Please help. I get the following error whenever I try to run Kalanyr's most recent version and cannot login.

Traceback (most recent call last):
File "F:\GOG\gogrepo.py", line 40, in <module>
import requests
ModuleNotFoundError: No module named 'requests'

F:\GOG is the folder I am running it from. I have tried fresh installs of Python 2.7 and 3.7 and got the same results. I used pip to download and install the modules needed. Any advice would be welcome. System specs below:

AMD FX-6300 @ 4.5Ghz
32 GB DDR3 RAM
512 GB Samsung 850 PRO SSD (C Drive Windows 10 April 2018 Update & Python are)
3TB Seagate HDD
Windows 10 Pro 1803 17134.165

Thanks in advance!

Edit: Got it working! I just had to install some unlisted prerequisite modules.
Post edited July 12, 2018 by MikhailP

[kohlrak@kohlrak-server gog]$ cat gogrepo.log | grep "changelog was updated" -B 2
12:33:52 | (231 / 559) fetching game details for master_of_magic...
12:33:57 | (232 / 559) fetching game details for master_of_orion...
12:34:11 | -> changelog was updated
--
12:50:01 | (427 / 559) fetching game details for system_shock_2...
12:50:08 | -> gog flagged this game as updated
12:50:08 | -> changelog was updated
12:50:08 | (428 / 559) fetching game details for tachyon_the_fringe...
12:50:11 | (429 / 559) fetching game details for tales_of_majeyal...
12:50:16 | -> changelog was updated
--
12:58:31 | (510 / 559) fetching game details for vampire_the_masquerade_bloodlines...
12:58:46 | -> gog flagged this game as updated
12:58:46 | -> changelog was updated
I didn't check master_of_orion, but I did check tales_of_majeyal:
changelog......:

<h4>Patch 1.5.10 (25 May 2018)</h4>
<ul>
<li>Zigur is now always present on the map but forbids entry to any magic user.</li>
<li>Fix addons & DLCs auto updater</li>
<li>Updated registration process for new players</li>
Not a huge update, but when minor changes are getting marked for update...

And what about games that might upload to gog but don't keep proper changelogs ('cause, no one would ever do that)? Does gog properly mark them as updated, if they don't flag as updated for these bug fixes? Seems like full updates should be advised.
The Updated / updateonly command uses the same definition of updated as the GOG.com site, which is imperfect.

Same with New / skiponly (which is generally perfect except for DLC and Pre-ordered games).

Full updates will precisely sync your manifest with what GOG thinks is true for every file of every game you own (except that filtered out by eg language / OS / Galaxy installers)


Changelogs are only checked and notified after it's decided a game needs to have it's manifest redone, so non-updated manifests don't really have a bearing one way or the other.
avatar
mqstout: Download kept failing until I did login AND update again.
This sounds like your cookie ID or manifest got blacklisted or corrupted for some reason. That is one heck of a badly timed segfault.
Post edited July 13, 2018 by Kalanyr
avatar
Kalanyr: The Updated / updateonly command uses the same definition of updated as the GOG.com site, which is imperfect.

Same with New / skiponly (which is generally perfect except for DLC and Pre-ordered games).

Full updates will precisely sync your manifest with what GOG thinks is true for every file of every game you own (except that filtered out by eg language / OS / Galaxy installers)

Changelogs are only checked and notified after it's decided a game needs to have it's manifest redone, so non-updated manifests don't really have a bearing one way or the other.
I'm more referring to the practice that GOG isn't flagging games as updated despite changing their changelog, showing that full updates are important, even if we're getting a ton of timeout errors when we do it.
avatar
MikhailP: Please help. I get the following error whenever I try to run Kalanyr's most recent version and cannot login.

Traceback (most recent call last):
File "F:\GOG\gogrepo.py", line 40, in <module>
import requests
ModuleNotFoundError: No module named 'requests'

F:\GOG is the folder I am running it from. I have tried fresh installs of Python 2.7 and 3.7 and got the same results. I used pip to download and install the modules needed. Any advice would be welcome. System specs below:

AMD FX-6300 @ 4.5Ghz
32 GB DDR3 RAM
512 GB Samsung 850 PRO SSD (C Drive Windows 10 April 2018 Update & Python are)
3TB Seagate HDD
Windows 10 Pro 1803 17134.165

Thanks in advance!

Edit: Got it working! I just had to install some unlisted prerequisite modules.
I'm trying to switch to Kalanyr's version and I'm running into the exact same issue. any guidance on what prereqs need to be installed? (and how?)

edit: the prerequs in question are the python "requests" module, and the "pyopenssl" module. the latter can be installed by typing "pip install pyopenssl" in your command line. however, I am now getting an error when attempting download with any version of kalanyr's script.
L:\BACKUP\GAMES\GOG_INSTALLERS>gogrepo.py download
19:02:37 | loading local manifest...
19:02:37 | scanning manifest for renames...
19:02:37 | fatal...
Traceback (most recent call last):
File "L:\BACKUP\GAMES\GOG_INSTALLERS\gogrepo.py", line 2342, in <module>
main(process_argv(sys.argv))
File "L:\BACKUP\GAMES\GOG_INSTALLERS\gogrepo.py", line 2275, in main
cmd_download(args.savedir, args.skipextras, args.skipids, args.dryrun, args.ids,args.os,args.lang,args.skipgalaxy,args.skipstandalone,args.skipshared)
File "L:\BACKUP\GAMES\GOG_INSTALLERS\gogrepo.py", line 1395, in cmd_download
handle_game_renames(savedir,items,dryrun)
File "L:\BACKUP\GAMES\GOG_INSTALLERS\gogrepo.py", line 483, in handle_game_renames
for item in game.downloads+game.galaxyDownloads+game.sharedDownloads+game.extras:
File "L:\BACKUP\GAMES\GOG_INSTALLERS\gogrepo.py", line 264, in __getattr__
raise AttributeError(key)
AttributeError: galaxyDownloads
I get similar (though non-fatal) errors on verification, and on update.
Post edited July 15, 2018 by DoctorDevice
avatar
MikhailP: Please help. I get the following error whenever I try to run Kalanyr's most recent version and cannot login.

Traceback (most recent call last):
File "F:\GOG\gogrepo.py", line 40, in <module>
import requests
ModuleNotFoundError: No module named 'requests'

F:\GOG is the folder I am running it from. I have tried fresh installs of Python 2.7 and 3.7 and got the same results. I used pip to download and install the modules needed. Any advice would be welcome. System specs below:

AMD FX-6300 @ 4.5Ghz
32 GB DDR3 RAM
512 GB Samsung 850 PRO SSD (C Drive Windows 10 April 2018 Update & Python are)
3TB Seagate HDD
Windows 10 Pro 1803 17134.165

Thanks in advance!

Edit: Got it working! I just had to install some unlisted prerequisite modules.
avatar
DoctorDevice: I'm trying to switch to Kalanyr's version and I'm running into the exact same issue. any guidance on what prereqs need to be installed? (and how?)

edit: the prerequs in question are the python "requests" module, and the "pyopenssl" module. the latter can be installed by typing "pip install pyopenssl" in your command line. however, I am now getting an error when attempting download with any version of kalanyr's script.

L:\BACKUP\GAMES\GOG_INSTALLERS>gogrepo.py download
19:02:37 | loading local manifest...
19:02:37 | scanning manifest for renames...
19:02:37 | fatal...
Traceback (most recent call last):
File "L:\BACKUP\GAMES\GOG_INSTALLERS\gogrepo.py", line 2342, in <module>
main(process_argv(sys.argv))
File "L:\BACKUP\GAMES\GOG_INSTALLERS\gogrepo.py", line 2275, in main
cmd_download(args.savedir, args.skipextras, args.skipids, args.dryrun, args.ids,args.os,args.lang,args.skipgalaxy,args.skipstandalone,args.skipshared)
File "L:\BACKUP\GAMES\GOG_INSTALLERS\gogrepo.py", line 1395, in cmd_download
handle_game_renames(savedir,items,dryrun)
File "L:\BACKUP\GAMES\GOG_INSTALLERS\gogrepo.py", line 483, in handle_game_renames
for item in game.downloads+game.galaxyDownloads+game.sharedDownloads+game.extras:
File "L:\BACKUP\GAMES\GOG_INSTALLERS\gogrepo.py", line 264, in __getattr__
raise AttributeError(key)
AttributeError: galaxyDownloads
avatar
DoctorDevice: I get similar (though non-fatal) errors on verification, and on update.
Blargh. That's an error on my part where I've forgotten to clean up an enry somewhere to account for old manifest entries not having galaxyDownloads. I'll take a look through tonight and see if I can spot the problem.
I think I've identified the issue.

Could you make a backup copy of your current manifest and then try the following:

Would I be correct in thinking that if you do a full manifest update instead of new / updateonly that this issue is fixed / doesn't happen ?

(It looks like I forgot to account for the possibility of coming into identifying renamed games/files with a manifest that's only partially in my forks format. Somedays I think I should just zap any old menifest into the new format as a first step. I'll push an update to the dev branch as soon as I can sit down and do some work (I'm travelling in Europe at the moment so my programming time is limited)) that should fix this, and if you could verify with the old manifest that would be great.
I have no real facility whatsoever to test this but this is a dev branch potential push that should theoertically fix the issue:

https://www.dropbox.com/s/2v8titgk325ebs9/gogrepoc.py?dl=0

Syntax and behaviour is a tad different from the current main fork , but the built in help and error messages should hopefully get you through if you don't use the dev fork.

Feel free to ask here to and I'll answer.


I probably won't roll this back into main, I'll be back in two week and then I'll dedicate a week or two to rolling the new version out properly.
This is a link to a temp version of the current non-dev build that should fix the issue in it too (let me know if it doesn't).

It's also untested. I'm travelling and have no access to an old style manifest to test with or a repo to test on.

https://www.dropbox.com/s/9ha8b8hgni4ztp7/gogrepo.py?dl=0
Post edited July 15, 2018 by Kalanyr