It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(4233 posts)(4233 posts)
(4233 posts)
Pages:
1
50
100
150
200
250
277
278
279
280
281
283
This is my favourite topic
Posted April 12, 2025
Well, speaking of a manually downloaded collection, I've identified a problem (at least for me).

I've chosen the approach to make symbolic links from my old folders to the "GOG names" in the "repository directory".

This is working rather well, and I want to keep it this way, because I've structured games like this:

…/rpg/TES/Arena
…/rpg/TES/Daggerfall
…

I've also stuff from elsewhere, like solutions and modifications for games I have on GOG, which I like to keep in the same folders rather than elsewhere.

This means, of course, that I can't use the "clean" command, which I don't want to anyway, because as a Linux user, it's not wise to only keep the most recent version of everything, they might break it at any time (example: no Linux version exists, they abandon support for an old Windows version in favour of Windows 11, while Wine works only flawlessly with the game when set to that old version).

The "implicit clean" whenever GOG changes the contents of a file, but keeps the old file name, is nothing to worry about: such events aren't very frequent, I can decide individually what to do (rename the old files, delete them, whatever seems approriate). Now the problem: there's one exception: if instructed to download images as well (-covers,…), it seems to do what the "clean" command normally does: it throws away not only images of the same name where the content has changed, but also those "not belonging there".

This is a problem whenever a game and an "extra" package are kept in the same folder: when using the "normal clean" on that, everything will be moved to "!orphaned".

Example:
2 symlinks point from 2 "GOG entities" to the same folder:
gog-repository/game -> /path/to/Game
gog-repository/game-goodies -> /path/to/Game

In "/path/to/Game" are:
goodies.zip (from "game-goodies" in the Manifest)
game-setup.sh (from "game" in the Manifest)
game-setup.exe (from "game" in the Manifest)

If you call "clean" on that, it will move the installation files to "!orphaned/game-goodies" when working on "game-goodies" and "goodies.zip" to "!orphaned/game" when working on "game".

This is not a problem generally, because I won't use the "clean" command, but in case of an "!images" folder existing, that's rather annoying, because images belongig to another GOG package will be moved away, just because GOG has changed some installation file's MD5/size, keeping its name.

At least this is what I believe has been happening here.

(Of course there's a workaround: I could generally keep my structure, but would have to create a subfolder underneath exclusively for each "GOG folder name", but I'd prefer to avoid this otherwise uneccessary work.)

Any ideas?
Post edited April 12, 2025 by ChFra
Posted April 12, 2025
I could add change the image/cover update options to match verify so that it can clean/delete/ignore , would just require adding ignore, since the other two behaviours are already present. I see some potential edge case problems but none worse than ignoring a file that failed verify.
Posted April 15, 2025
avatar
mrkgnao: Feature request:

GOG seems to be adding more and more meaningless entries to some games. For example:

18:47:22 | no known filename for "blood_west (Blood West)"
18:47:33 | no known filename for "hacknet (Labyrinths DLC)"
18:47:33 | no known filename for "iratus_lord_of_the_dead (additional skins for all minions)"
18:47:33 | no known filename for "jagged_alliance_2_wildfire (Jagged Alliance 2: Wildfire)"
18:47:33 | no known filename for "omerta_city_of_gangsters (Omerta - City of Gangsters - The Bulgarian Colossus DLC)"
18:47:33 | no known filename for "omerta_city_of_gangsters (The Arms Industry DLC)"
18:47:33 | no known filename for "omerta_city_of_gangsters (Damsel In Distress DLC)"
18:47:33 | no known filename for "omerta_city_of_gangsters (The Con Artist DLC)"
18:47:33 | no known filename for "shadow_tactics_blades_of_the_shogun (Shadow Tactics: Blades of the Shogun)"
18:47:46 | no known filename for "stray_gods_the_roleplaying_musical (Stray Gods: The Roleplaying Musical)"
18:47:46 | no known filename for "warhammer_skulls_2022_digital_goodie_pack_goodies (Warhammer Chaos Gate)"

Perhaps gogrepoc can somehow learn to ignore these kind of entries.
avatar
Kalanyr: Should be mostly ignoring them already, I assume this is from verify ? I don't think I've checked it yet.
I finally found some time to install the new version. Thank you for fixing this!
Post edited April 15, 2025 by mrkgnao
Posted April 15, 2025
I've now been testing the "old current" version (before you movedd the "clean warning" from "import" to "verify") for a few days nowm, and can confirm that I didn't encounter any issues apart from the one below (reported in my last post).

I tested at least the following combinations:
update -resumemode resume -md5xmls -installers standalone
update -resumemode resume -md5xmls -installers standalone -standard
update -resumemode resume -md5xmls -installers standalone -strictverify -strictdupe -strictextrasupdate -full
verify $FOLDER -skipgalaxy -os … -lang … -clean
verify $FOLDER -skipgalaxy -os … -lang … -clean -forceverify
download $FOLDER -skipgalaxy -os … -lang … -ids …
download $FOLDER -skipgalaxy -os … -lang … -covers -backgrounds -ids …

(From a wrapper script I can call from everywhere to do more easily what I usually would do. The -ids is for handling only preexisting "game entries" I keep in $FOLDER:
cd $FOLDER
for GAME in *
do
cd ..
python gogrepoc.py download … -ids "$GAME"
cd $FOLDER
done
…

This way I can temporarily move symlinks elsewhere to exclude games from being actually updated, and can keep my old folder structure from before using gogrepoc.py)

Those didn't do anything I didn't expect so far with one exception I report below.

avatar
Kalanyr: I could add change the image/cover update options to match verify so that it can clean/delete/ignore , would just require adding ignore, since the other two behaviours are already present. I see some potential edge case problems but none worse than ignoring a file that failed verify.
I can't currently imagine those edge cases yet, but would be happy if you considered implementing that.

At last the problem I mentioned above:

Since there's no option to limit the bandwidth the downloads consume, I'll have to interrupt the process occasionally. I do this by pressing "Ctrl-Z" on the console where my script is running in the foreground.
Wenever the time before resuming is too long, gogrepoc.py will run into timeouts for the currently downloaded set of files, skipping them altogether, moving on to the remaining files. This results in those files remaining partially donloaded and still missing in the destination folders when gogrepoc.py ends.
This is not a big problem, because it can be restarted, and I always see this happening, becausse I interrupt and resume manually, but it would be nice to have those downloads not "removed from the queue", but retried immediately (or later, at least before it quits).
However, I see the potential problem when implementing this that something could be wrong with a file on GOG's side causing gogrepoc.py to run endlessly trying to "resume" again and again.

Thank you for writing and maintaining "gogrepoc.py"!
Posted April 15, 2025
Greetings and a big thank you for this great script!

I used it on my WIN10 with no problem. Now on WIN1124H2 some problems have arisen.

I solved it with using python 2+ since 3+ gave me file writing errors.

Now the update and download, clean and trash commands work flawlessly, but the verify command is still giving me problems that I cannot solve and can't find the solution to on stackoverflow. This is what I get in my window:

20:01:34 | scanning manifest for renames...
20:01:34 | fatal...
Traceback (most recent call last):
File "J:\GOG\gogrepo.py", line 400, in __getattr__
return self[key]
~~~~^^^^^
KeyError: 'unreleased'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "J:\GOG\gogrepo.py", line 4263, in <module>
main(process_argv(sys.argv))
~~~~^^^^^^^^^^^^^^^^^^^^^^^^
File "J:\GOG\gogrepo.py", line 4060, in main
cmd_verify(args.gamedir, args.skipextras,args.skipids,check_md5, check_filesize, check_zips, args.delete,not args.noclean,args.ids, args.os, args.lang,args.skipgalaxy,args.skipstandalone,args.skipshared, args.skipfiles, args.forceverify,args.permissivechangeclear)
~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "J:\GOG\gogrepo.py", line 3632, in cmd_verify
if itm.unreleased:
^^^^^^^^^^^^^^
File "J:\GOG\gogrepo.py", line 402, in __getattr__
raise AttributeError(key)
AttributeError: unreleased

Does anybody know what could be the problem?

The command I try to write is "verify -skipextras"
Post edited April 15, 2025 by Modelsson
Posted April 15, 2025
avatar
Modelsson: File "J:\GOG\gogrepo.py", line 3632, in cmd_verify
if itm.unreleased:
^^^^^^^^^^^^^^
File "J:\GOG\gogrepo.py", line 402, in __getattr__
raise AttributeError(key)
AttributeError: unreleased

Does anybody know what could be the problem?

The command I try to write is "verify -skipextras"
Lines 3632+3633 read here (without the indentation):
===================
if itm.unreleased:
continue
===================
If I understand Python correctly, there should no exception be raised.
Are you running the current version?
Post edited April 15, 2025 by ChFra
Posted April 15, 2025
@Kalanyr

I'm currently wondering about the reason to download 4 files at a time, at least for me that doesn't speed up anything, having the disadvantage of multiple partial files on crashes/aborts. Maybe for fibre users (if GOG limits "throughput per connection")?
Post edited April 15, 2025 by ChFra
Posted April 16, 2025
avatar
ChFra: @Kalanyr

I'm currently wondering about the reason to download 4 files at a time, at least for me that doesn't speed up anything, having the disadvantage of multiple partial files on crashes/aborts. Maybe for fibre users (if GOG limits "throughput per connection")?
I have a 350Mb/s connection. When I download four or more files, it uses the entire bandwidth. When I download only one/two/three files, it is limited to about one/two/three quarters of the bandwidth. Someone (GOG?, python?, other?) seems to be limiting a single download to ~8MB/s.
Posted April 16, 2025
avatar
ChFra: @Kalanyr

I'm currently wondering about the reason to download 4 files at a time, at least for me that doesn't speed up anything, having the disadvantage of multiple partial files on crashes/aborts. Maybe for fibre users (if GOG limits "throughput per connection")?
avatar
mrkgnao: I have a 350Mb/s connection. When I download four or more files, it uses the entire bandwidth. When I download only one/two/three files, it is limited to about one/two/three quarters of the bandwidth. Someone (GOG?, python?, other?) seems to be limiting a single download to ~8MB/s.
GOG seems to have inconsistent settings for this depending on what your downloading (Windows and Extras usually download faster than Linux / Mac ) and where you are on the CDN (for me individual files don't seem to be limited but the total is capped at ~18 MB / sec for Windows ) so it's a case where YMMV depending, so you should tweak the number of simultaneous downloads to something that works for you.
Posted April 16, 2025
avatar
mrkgnao: I have a 350Mb/s connection. When I download four or more files, it uses the entire bandwidth. When I download only one/two/three files, it is limited to about one/two/three quarters of the bandwidth. Someone (GOG?, python?, other?) seems to be limiting a single download to ~8MB/s.
avatar
Kalanyr: GOG seems to have inconsistent settings for this depending on what your downloading (Windows and Extras usually download faster than Linux / Mac ) and where you are on the CDN (for me individual files don't seem to be limited but the total is capped at ~18 MB / sec for Windows ) so it's a case where YMMV depending, so you should tweak the number of simultaneous downloads to something that works for you.
I see, so this must remain as is in order to not hamper users with fast connections; as I've got a 10MBit/s half-duplex cable between me and the internet, I won't see any difference in overall performance, I think, no matter how many files are fetched at the same time, so I'd be better off with one at a time.

Please correct me, if I that's incorrect, but to change this, there is no command line parameter, I'll have to change by hand: "HTTP_GAME_DOWNLOADER_THREADS = 4" from "4" to "1"?
Posted April 16, 2025
avatar
Kalanyr: GOG seems to have inconsistent settings for this depending on what your downloading (Windows and Extras usually download faster than Linux / Mac ) and where you are on the CDN (for me individual files don't seem to be limited but the total is capped at ~18 MB / sec for Windows ) so it's a case where YMMV depending, so you should tweak the number of simultaneous downloads to something that works for you.
avatar
ChFra: I see, so this must remain as is in order to not hamper users with fast connections; as I've got a 10MBit/s half-duplex cable between me and the internet, I won't see any difference in overall performance, I think, no matter how many files are fetched at the same time, so I'd be better off with one at a time.

Please correct me, if I that's incorrect, but to change this, there is no command line parameter, I'll have to change by hand: "HTTP_GAME_DOWNLOADER_THREADS = 4" from "4" to "1"?
Yup. I should probably provide a way to set a default configuration at some point .
Posted April 16, 2025
avatar
ChFra: Please correct me, if I that's incorrect, but to change this, there is no command line parameter, I'll have to change by hand: "HTTP_GAME_DOWNLOADER_THREADS = 4" from "4" to "1"?
Maybe not a biggie, but it might still be a better idea to keep it at least at e.g. 2 so that if that one file somehow stalls or there is pause when one file has finished and another is starting, there'd be another file also downloading and using the bandwidth also on those instances.

I have a similar experience as Kalanyr with my CDN, ie. it doesn't make much difference how many files I download at the same time from GOG, the total is still usually capped to e.g. 19-20 MBytes/sec, no matter if I download 1 or 4 files at the same time. Sometimes it is faster though, up to 34 MBytes/sec or so (my theoretical maximum is around 75 MBytes/sec I think).

Still, I use 2 or 3 download threads, for the reason I mentioned earlier. It just tends to give a more predictable and even speed overall, compared to downloading only one file at a time.
Post edited April 16, 2025 by timppu
Posted April 16, 2025
Weird errors occur (times out, and as well for all 4 retries):

22:46:07 | request failed: 403 Client Error: Forbidden for url: https://cdn.gog.com/secure/baldurs_gate_3_goodie_pack/bg3_wallpaper_01.zip?1223fb43868c5f174d699acb523ffcf6bf0b4bf7e9c953140a9bc129d483be63e9324bc47b747dfcbe6605086d1cbb498242ff6e78129d92c1fd4666de11654a48b2daa22c6fa1c7c653741e82f04852280d6717daa25bd1a616602f68954c9c4af6a22d9fdc905adac714465597d3cf0bba67c97033c3d92be4ec48b7a2ef1b2c&amp;fileExtForIe=.exe. will not retry.

I did not notice that before.

At the monent, it happens frequently when doing a "complete" update with "-full".

Without "-full" as well, happens always when it tries to fetch the same URLs.

SEEMS TO BE A GOG PROBLEM! Has apparently gone round about 22:00 UTC. However, retrying for minutes on 403/404 seems unreasonable, those normally won't go away in a few seconds other than 500/503 where the server can often recover from.
Post edited April 17, 2025 by ChFra
Posted April 17, 2025
avatar
ChFra: Weird errors occur (times out, and as well for all 4 retries):

22:46:07 | request failed: 403 Client Error: Forbidden for url: https://cdn.gog.com/secure/baldurs_gate_3_goodie_pack/bg3_wallpaper_01.zip?1223fb43868c5f174d699acb523ffcf6bf0b4bf7e9c953140a9bc129d483be63e9324bc47b747dfcbe6605086d1cbb498242ff6e78129d92c1fd4666de11654a48b2daa22c6fa1c7c653741e82f04852280d6717daa25bd1a616602f68954c9c4af6a22d9fdc905adac714465597d3cf0bba67c97033c3d92be4ec48b7a2ef1b2c&amp;fileExtForIe=.exe. will not retry.

I did not notice that before.

At the monent, it happens frequently when doing a "complete" update with "-full".

Without "-full" as well, happens always when it tries to fetch the same URLs.

SEEMS TO BE A GOG PROBLEM! Has apparently gone round about 22:00 UTC. However, retrying for minutes on 403/404 seems unreasonable, those normally won't go away in a few seconds other than 500/503 where the server can often recover from.
403 is one of the ones where GOG uses the wrong response code sometimes, they use it when the server is overloaded (which is not appropriate, there's a defined error code for that that also includes a specification of how long to wait until retry ) but I have to treat 403 as 403 just in case something generates a 403 for real ( like you getting singed out for some reason).
Post edited April 17, 2025 by Kalanyr
Posted April 17, 2025
avatar
Modelsson: File "J:\GOG\gogrepo.py", line 3632, in cmd_verify
if itm.unreleased:
^^^^^^^^^^^^^^
File "J:\GOG\gogrepo.py", line 402, in __getattr__
raise AttributeError(key)
AttributeError: unreleased

Does anybody know what could be the problem?

The command I try to write is "verify -skipextras"
avatar
ChFra: Lines 3632+3633 read here (without the indentation):
===================
if itm.unreleased:
continue
===================
If I understand Python correctly, there should no exception be raised.
Are you running the current version?
Thanks for your reply.

I'm runing 3.13.3 but in Win11 App execution aliases I have "python3.exe" off and only "python.exe" on.

If I understand you correctly, I need to add your code to the lines you suggested into the py file? I am using the latest version of gogrepoc. I copied the raw code and made a py file myself.

And I can confirm the almost 8 MB/S speed of download. I thought GOG just lowered it. Though I can download from GOG directly at 50 MB/s
Post edited April 17, 2025 by Modelsson
Pages:
1
50
100
150
200
250
277
278
279
280
281
283
This is my favourite topic
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(4233 posts)(4233 posts)
(4233 posts)