It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(4233 posts)(4233 posts)
(4233 posts)
Pages:
1
9
10
11
12
13
50
100
150
200
250
283
This is my favourite topic
Posted August 27, 2015
I went back and checked the manifest and it says 3d , so download* seem to be case agnostic while clean is case sensitive. Unless the weirdness with the case is just causing the download / verify to fail silently ?

I know that at the GUI level and using the standard window GUI API Window's ignores attempts to rename that only change case, so can you rename from 3D to 3d using python , if you make it detect case ? I guess if you detect that the problem is only case you can do something like rename it to something temporary before swapping back with corrected case.

*and maybe verify ?
Post edited August 27, 2015 by Kalanyr
Posted August 27, 2015
avatar
timppu: Iron Storm seems to have an interesting issue. Not sure if this is a problem with gogrepo script, or the GOG database.

I noticed the !info.txt file for Iron Storm doesn't contain any game items (installers or patches), only extras. On the GOG account page, I notice that for Iron Storm there is an installer, but for some reason the only selectable language is "POLSKI". No English. According to the Iron Storm gamecard though, it should support various languages:
avatar
woolymethodman: 99% this is GOG side problem.

don't have that game to check. Indeed it depends on the language from the drop down. If POLSKI is the only language, then that's a bug on GOG side.
GOG seems to have fixed this now. :) Now the download page shows English as the default download language, and Polski as another. I haven't tried it but I presume the issue is now fixed for gogrepo.

I'll have to try to hunt down somehow if the same problem exists with some other games I have. I still need to figure out an easy way to check it (without having to check each game one by one), somehow with either the !info.txt files that gogrepo creates, or from the gogrepo update or verify output, which titles are missing setup_* files. At least checking how many English games have a setup_*.exe file should be quite straightforward from the gogrepo output...
Post edited August 27, 2015 by timppu
Posted August 27, 2015
avatar
woolymethodman: 99% this is GOG side problem.

don't have that game to check. Indeed it depends on the language from the drop down. If POLSKI is the only language, then that's a bug on GOG side.
avatar
timppu: GOG seems to have fixed this now. :) Now the download page shows English as the default download language, and Polski as another. I haven't tried it but I presume the issue is now fixed for gogrepo.

I'll have to try to hunt down somehow if the same problem exists with some other games I have. I still need to figure out an easy way to check it (without having to check each game one by one), somehow with either the !info.txt files that gogrepo creates, or from the gogrepo update or verify output, which titles are missing setup_* files. At least checking how many English games have a setup_*.exe file should be quite straightforward from the gogrepo output...
Cool. Maybe I can have a warning printed in log if a game is found with no "english" installer files (if english was specified). I feel english is a special case here as I assume every game on GOG should have a file for "english".
Posted August 30, 2015
I am really interested in using this but 'm having a hell of a time getting it to work in windows.

e:\Gamez\GoG>gogrepo.py login
File "E:\Gamez\GoG\gogrepo.py", line 4
<!DOCTYPE html>
^
SyntaxError: invalid syntax
this is the error i get

I have installed the html5lib via command C:\Python27\Scripts>pip install html5lib it downloaded and said succeed.

My next step is to get a VM of linux mint that has read/write access to my windows gog folder and use that.. which sounds like a PITA :p
mint@mint-virtual-machine /mnt/hgfs/GoG $ python gogrepo.py login
File "gogrepo.py", line 4
<!DOCTYPE html>
^
SyntaxError: invalid syntax
mint@mint-virtual-machine /mnt/hgfs/GoG $
And here is the error in my linux box... and yes i installed html5lib :(
Post edited August 30, 2015 by Starkrun
Posted August 30, 2015
avatar
Starkrun: I am really interested in using this but 'm having a hell of a time getting it to work in windows.

e:\Gamez\GoG>gogrepo.py login
File "E:\Gamez\GoG\gogrepo.py", line 4
<!DOCTYPE html>
^
SyntaxError: invalid syntax
avatar
Starkrun:
Sounds like you didn't download the gogrepo.py file correctly (there is no <!DOCTYPE html> anywhere in the script)

Try this:
https://raw.githubusercontent.com/eddie3/gogrepo/master/gogrepo.py
Posted August 30, 2015
avatar
woolymethodman: https://raw.githubusercontent.com/eddie3/gogrepo/master/gogrepo.py
Thank you that was it! i did a rightclik save as from the link on github :/

here is link i used: https://github.com/eddie3/gogrepo/blob/master/gogrepo.py

which now i see my issue... i'll grab the RAW next time. This was my first experience with GitHub.
Post edited August 30, 2015 by Starkrun
Posted August 30, 2015
Coming soon! Update command will show what was changed for each game (just the more interesting fields).

eg.

00:45:07 | ( 1 / 868) fetching game details for 1954_alcatraz...
00:46:07 | -> gog flagged this game as updated
00:46:07 | -> title change "1954_alcatraz" -> "1954_alcatraz_classic"
00:46:07 | -> long title change "1954 Alcatraz" -> "1954 Alcatraz Classic"
00:46:07 | -> game changelog was updated
00:46:07 | -> rating change 39 -> 30
00:46:07 | -> game has new serial key!

If a title change is detected, the script will rename the local game folder to adjust to the new name. This will avoid re-download the game again (eg. witcher_3 -> witcher_3_game).
Posted August 30, 2015
avatar
woolymethodman: https://raw.githubusercontent.com/eddie3/gogrepo/master/gogrepo.py
avatar
Starkrun: Thank you that was it! i did a rightclik save as from the link on github :/

here is link i used: https://github.com/eddie3/gogrepo/blob/master/gogrepo.py

which now i see my issue... i'll grab the RAW next time. This was my first experience with GitHub.
If you look on the main page at the right:
https://github.com/eddie3/gogrepo
There is a button "Download Zip"
Posted August 30, 2015
Your are awesome! Thanks for this!!
Posted August 30, 2015
Import just did a massive md5 check with no move. so i ended up using the Backup command and then using winmerge to delete the source files.

Wanted to let you know import didnot work :(
PS E:\Gamez\GoG> python gogrepo.py import E:\Gamez\GoG E:\Gamez\GoGclean
05:48:48 | loading local manifest...
05:48:48 | collecting md5 data out of the manifest
05:48:48 | searching for files within 'E:\Gamez\GoG'
05:48:49 | comparing md5 file hashes
05:48:49 | calculating md5 for 'directx_Jun2010_redist.exe'
05:48:49 | calculating md5 for 'gog-cookies.dat'
05:48:49 | calculating md5 for 'gog-manifest.dat'
05:48:49 | calculating md5 for 'gogrepo.py'
05:48:49 | calculating md5 for 'Gothic 1-2 windows 8.odt'
05:48:49 | calculating md5 for 'Setup_Downloader_3.0.51b.exe'
05:48:49 | calculating md5 for 'Thumbs.db'
05:48:49 | calculating md5 for 'setup_advent_rising-1.bin'
...
06:37:00 | calculating md5 for 'setup_xiii_2.0.0.11.exe'
06:37:00 | --
06:37:00 | total time: 0:48:11.799000
06:37:00 | exiting...
Posted August 30, 2015
avatar
Lin545: Your are awesome! Thanks for this!!
Thanks, and enjoy!
avatar
Starkrun: Import just did a massive md5 check with no move. so i ended up using the Backup command and then using winmerge to delete the source files.

Wanted to let you know import didnot work :(
Thanks for reporting this. I'll take a look.
Post edited August 30, 2015 by woolymethodman
Posted September 06, 2015
avatar
Starkrun: Import just did a massive md5 check with no move. so i ended up using the Backup command and then using winmerge to delete the source files.

Wanted to let you know import didnot work :(
That has been bugging me: what exactly is the difference between import and backup commands (in case I want to create a clean GOG set from an old set; yeah I know there is now the clean command too...)? The readme suggests using backup for that, so what is the purpose for import?

Is the difference that import will find and even rename relevant file and directory names to correspond to the new manifest file, while backup will look only for correctly named files and directories in the source? If so, then I presume import would be a preferable option, as it would handle the constant game directory name changes by GOG as well, without needing to redownload anything?

Why did you need to use winmerge to delete the old files? Why not just delete them after the backup command has done its job?
Post edited September 06, 2015 by timppu
Posted September 06, 2015
avatar
timppu: That has been bugging me: what exactly is the difference between import and backup commands (in case I want to create a clean GOG set from an old set; yeah I know there is now the clean command too...)? The readme suggests using backup for that, so what is the purpose for import?

Why did you need to use winmerge to delete the old files? Why not just delete them after the backup command has done its job?
import looks like it was supposed to move only the valid files to a new location.. clean moves what import would have skipped (or the invalid files) to a folder called "!orphaned" its pretty much the same thing. One makes a new folder outside of the container object and the other places it in the container.

As for winmerge, which I used a bash commands instead comparing one folder to the other. Checked if the destination contained the same folder from the source if it did then it deleted the source file/folder.

From now on i'll be using clean as i dont mind the "!orphaned" folder being generated. Now if GOG can fix the bad downloads and broken md5's they have for there games :p
Post edited September 06, 2015 by Starkrun
Posted September 06, 2015
avatar
Starkrun: import looks like it was supposed to move only the valid files to a new location.. clean moves what import would have skipped (or the invalid files) to a folder called "!orphaned" its pretty much the same thing. One makes a new folder outside of the container object and the other places it in the container.
If GOG has merely changed the game directory name but the files are the same, what happens then? E.g. how they changed the_witcher_3_wild_hunt => the_witcher_3_wild_hunt_game?

Does clean command move the whole the_witcher_3_wild_hunt (with its files) into the orphaned directory? I guess that is better than nothing, then you can at least identify that the new manifest file considers them obsolete files (or directory), and check manually why that is. Then just copy the whole TW3 directory back to the main directory, and rename it according to the new format. Then it is not considered as an orphaned set of files, I guess.

From the readme file, I was under the impression that the import command would have actually moved the valid files from a source the_witcher_3_wild_hunt directory to the new the_witcher_3_wild_hunt_game directory. In practice renaming the old directory name to the new one. That would have been a nice feature. As it is now, unless you change the directory name manually, the same files will be redownloaded.

Ps. How widespread is the "wrong md5 values" issue? Have they been reported to GOG? At least they changed the Iron Storm default language to the correct one when I reported it, that also caused a problem to gogrepo as it couldn't find the English language version of the game, as it was marked as "Polski".
Post edited September 06, 2015 by timppu
Posted September 06, 2015
import - this is for importing any games you have from external sources (gogdownloader, manually dl/renamed, not structured). It limits to md5 file matches for importing files. zip files don't have md5 so those are skipped. It's basically a safe way to import "as much as i can" from some random backup you may have that doesn't follow normal directory structure. Then you run download command to get whatever remaining files (extras, files without md5, etc) for a perfect collection.

backup - for backing up a repo made by gogrepo that follows the directory structure. This only matches <game>\<game_file> format and will include zip files (since size and dir/filename will match). This is for backing up your gogrepo collection to an off storage external drive over whatever. It doesn't copy files/folders that don't match the manifest.

clean - for removing files that do not match <game>\<game_file>. The purpose of this is to clean up/remove/orphan files that are considered OLD now (old patches from GOG, files etc). This is just to save you space, files are moved to !orphaned folder and you can decide if you want to delete them or not.

So basically use import to bring files from some random folder of games. Use backup to make a replica of your gogrepo collection. Use clean to remove older patches/files of your gogrepo collection.

I'll eventually add more info in the readme.

I don't think gog officially supports supplying MD5 info.. but they do fix them sometimes (more often than not). It's not a big deal that some files do not have MD5, it's just nice to have for verifying. I can probably pretty up the error message to not show the 404 and just make a info print that no MD5 info is available for the file.

I have a fix in works to detect GOG game title changes so folder would be renamed during 'update' so it will save you a re-download. Most commands like import/backup/clean use the local manifest for doing matches etc, so witcher3 -> witcher3_game wouldn't be an issue since witcher3_game is known folder.
Pages:
1
9
10
11
12
13
50
100
150
200
250
283
This is my favourite topic
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(4233 posts)(4233 posts)
(4233 posts)