It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
avatar
rtcvb32: Well taking a raw look, there's an awful lot of zeros. About what i expected.
avatar
clarry: That doesn't answer the question. What are the zeros there for, what do they represent? In all likelihood they are a part of something and not just "empty unused space." Zero is valid data too! And it happens to be very common data.
Zeros near other values certainly, but by itself generally not.

I am assuming an object has something like 100 properties, when those properties are blank it's 0. Putting all of them there whether they are used or not might or might not work.

avatar
clarry: Anyway, turns out I own the game and decided to take a look inside. While reverse engineering that binary format in 15 minutes is beyond my capabilities, I can make this observation: the file contains lots of recurring references to things like Chandelier, WallTorch, LightEffect, Teleporter, Destination, SetOfTiles*, SpikesSet*. Thousands of these. Which suggests to me that the file contains the entire scene with all its objects losslessly serialized.
Most likely named references to objects which is going to be in the sharedassets probably. Levels from what i see are every page and scene in the game, from the game menu and options, to each level in the game. As such each object is literally just a tile or object placed in space.

At least if my experience with Morrowind ESP files are anything to go by.

These details would have to include 3d points, scale, rotation, zoom, bounds checking model, and any other details it thinks it needs, including scripts (AI and otherwise). 99% of mapped landscape/level layout are going to be static tiles/blocks which requires no interactions.

avatar
clarry: Level15 appears to contains approximately 10k of such objects. I took a quick peek at a playthrough of the game on YouTube, and while it is not very easy to get the sense of the levels' scale, I can tell that they are quite large and can easily consist of that many tiles (and objects & effects contained within said tiles).

If all objects with all their properties and relations and references to assets are serialized in binary, the size looks quite sensible. A few kilobytes per object times 10k is how you get to a few tens of megabytes.
Yes, but you are still getting 200-300 0's in a row, which is why you'd go from 62Mb to 1.5Mb when compressing.

avatar
clarry: While I personally would not implement a game this way, I don't see anything wrong with it per se. The zeroes would be part of these serializations -- possibly including any padding bytes, if the goal is to make these files fast and easy to read & write without any marshalling (i.e. just copy the blob -- possibly even mmap the file and start using the memory as-is!).

Disk space is cheap and losslessly dumping objects in and out is fast and simple. Developer time is *not* cheap and I can see why someone would not spend it optimizing the data files, especially given how well entropy coding compresses it for distribution anyway.
Space may be cheaper now; but extracting a 50Mb archive to 530Mb is still disconcerting, especially if it's say an SSD with limited writes.

Now if Unity had done what a number of engines do which is allow you to access a zip file, then the 5-10% space lost due to inefficiency vs say 7z is insignificant.

Sorry, Space and efficiency is one of my big OCD's.

Ever written your own huffman code, store and load it and compress/decompress data on the fly?
avatar
clarry: Nope. But I have reverse engineered data files in games, including written a decompressor for the RLE compression in Killing Time and used that to uncompress & extract all the data files out of the archive.

Don't see the need to write my own huffman implementation given that there are plenty of free & open source implementations out there (including easy to use libraries). Also, if I wanted entropy coding, I would not waste time on old huffman. Arithmetic coding is much better.
It can be fun to implement for fun (though the sliding window is much simpler), but storing any kind of data means you're creating some impromptu format that is unique and not recognized by any other software; at least that is what i was aiming towards. So the 'Unity internal format' that eric5h5 was scoffing me about... well you get the idea.

Regardless, Arithmetic coding is a terrible name for it, as it doesn't do math or arithmetic at all; Instead it just creates additional symbols that contain a hidden length element attached to it. So if you encoded Hello and Helol the compression on Helol wouldn't be useful at all while it still would try to store the data for 1 L and 2 L's (Unless it's smart and knows it never finds 2 l's in a row).

I'd think arithmetic coding could just multiply values together (and you modulus to restore them), with enough symbols together the sum is smaller than the individual symbols. (32^5 vs 26^5)
avatar
randomuser.833: Why don't you check what the guys from Unity have written on their own before making assumptions?
avatar
neumi5694: Well, after the update it says that no matter they count the number of different devices it was installed on, so 20 installations on one computer count as 1.

If they are legally allowed to charge for something that has already been done, that's a different thing.
But in case it is so ... If a game generates 200k per year, then the dev can afford to pay for future customers. And if he uses the Pro version, that's 1 million $ and 1 million players to begin with.
You claimed, that they won't count past installations.
And they will count future "installations".
Means, a customer can purchase ONE copy of the game and Unity opens its hand every single time that guy got a new device.
Desktop, Laptop, Steam Deck and after some years a new Desktop and Laptop?
Unity has to be payed 5 times, over several years, while the Dev got the money for one sold Licence one time?


Yeah, I know, Big Boss of Unity said, that you are stupid and it is your own fault, if you don't charge the people who buy your game with micro transactions
avatar
randomuser.833: Desktop, Laptop, Steam Deck and after some years a new Desktop and Laptop?
Unity has to be payed 5 times, over several years,
In the worst case, that's 0.6$ (0.15$ for each device)
If you are a pro customier, it can go down to 0.05$ (0.01$ for each device)


Yes, after several years that's really unfair. We all agree on that.
avatar
randomuser.833: Desktop, Laptop, Steam Deck and after some years a new Desktop and Laptop?
Unity has to be payed 5 times, over several years,
avatar
neumi5694: In the worst case, that's 0.6$ (0.15$ for each device)
If you are a pro customier, it can go down to 0.05$ (0.01$ for each device)

Yes, after several years that's really unfair. We all agree on that.
It is already unfair, that you can be charged basically unlimited times for a single licence you sold.
That simply doesn't work.
And I don't know anything where that is the case.

If they would come up with an aditional fee for every licence you sold, when you are over certain target numbers, the situation would be very different.
It would be something you can estimate and calculate.

Here, some day Unity will come at you and will tell you how many unique installation they think (think, not certainly know) your game got and how much you own them based on that.
And it doesn't sound as if they are interested in showing you their numbers, because they are estimations, calculations based on certain models they made up.

They make the numbers, you have to pay.
Where is it that way?
avatar
rtcvb32: Yes, but you are still getting 200-300 0's in a row, which is why you'd go from 62Mb to 1.5Mb when compressing.
Nope! I just counted all the runs of 0s. The longest run is 158 zeroes and there is only one of them in the file. The next longest is 135 bytes and there are 13 of them.

There are 6063995 zeroes among a total of 64222112 bytes. That means only 9.4% of the file are zeroes. Runs longer than 32 bytes account for only 1.56% of the file. Runs shorter than 10 bytes account for nearly half of all the zeroes in the file. For something in the 64 bit era, these are very short runs. Just one short field and the padding to get your object aligned on 8 byte boundary could lead to 10 bytes of zeroes!

If anything, I'm surprised there aren't more of them. It's not uncommon for games to use god objects that have tons of fields for just about every feature -- most of them would be nuls for most objects.

There's plenty of data in the file. It's just low entropy data, which is why entropy coding does a good job of compressing it. The size is not due to having tons of unused space.
Post edited September 16, 2023 by clarry
avatar
rtcvb32: Yes, but you are still getting 200-300 0's in a row, which is why you'd go from 62Mb to 1.5Mb when compressing.
avatar
clarry: Nope! I just counted all the runs of 0s. The longest run is 158 zeroes and there is only one of them in the file. The next longest is 135 bytes and there are 13 of them.
158 is still really long.

avatar
clarry: There are 6063995 zeroes among a total of 64222112 bytes. That means only 9.4% of the file are zeroes
10% is 0's... Yeah that's surprising, i figured it was closer or 50% or worse. But as level 15 is compressible to about 1Mb that's like 60:1. Just means there's more patterns.

avatar
clarry: If anything, I'm surprised there aren't more of them. It's not uncommon for games to use god objects that have tons of fields for just about every feature -- most of them would be nuls for most objects.
Ditto. Only time i usually get nearly as good a result is when making a blank drive for DosBox or whatnot as a drive image, then formatting it to some simple Fat. Though those compress from 520Mb to 252k (then down to like 3k inside another zip).

Glancing with od and seeing multiple blocks of * (next lines are identical) I'd figured it was more zeros too. Though if each duplicate object is identical (other than XYZ/rotation) that would explain why it's finding as many matches.
There is one more interesting nock-on effect from this - most indie VR games are made with Unity. Unity actually have very good VR support and development frame work (Apple, for example, is relyiing heavily on Unity). But many indie developers have now had a it of a backlash from what is seen as a predatory pricing model, and are considering alternatives. I know at least four developers who are shifting their VR games development from Unity to Unreal. Unreal has a decent VR framework, but it is arguably not as good as Unity's framework. What Unreal has, though, is a much better graphics engine with Unreal 5.3, ecpesially as both Nanite and Lumen works in VR as well. So if nothing else, this may see a shift from Unity being the main VR game engine to Unity. Off course, this may have some impact on Meta Quest, but the new Quest 3 cominng out should be able to deal with it.

It also comes more to light that Godot actually has a VR framework, so it is also likely that Godot will also be more used for VR development, and the more it is used, the more it is also going to be improved. So in a way, this might in the end prove positive for VR gameing, moving away from a very heavy reliance on one engine.
avatar
amok: There is one more interesting nock-on effect from this - most indie VR games are made with Unity. Unity actually have very good VR support and development frame work (Apple, for example, is relyiing heavily on Unity). But many indie developers have now had a it of a backlash from what is seen as a predatory pricing model, and are considering alternatives. I know at least four developers who are shifting their VR games development from Unity to Unreal. Unreal has a decent VR framework, but it is arguably not as good as Unity's framework. What Unreal has, though, is a much better graphics engine with Unreal 5.3, ecpesially as both Nanite and Lumen works in VR as well. So if nothing else, this may see a shift from Unity being the main VR game engine to Unity. Off course, this may have some impact on Meta Quest, but the new Quest 3 cominng out should be able to deal with it.

It also comes more to light that Godot actually has a VR framework, so it is also likely that Godot will also be more used for VR development, and the more it is used, the more it is also going to be improved. So in a way, this might in the end prove positive for VR gameing, moving away from a very heavy reliance on one engine.
Ummm look up the system requirements for each tier for Immortals under UE 5. No way are VR games going to play well with native res. under the UE5 framework.
avatar
rtcvb32: You've obviously never done programming before.
I'll let clarry answer that:

avatar
clarry: And it doesn't sound like you understood eric5h5's explanation at all.
Let's look at Army of Ruin, the most recent Unity game I bought. Leaving out the framework stuff, the Data folder (which accounts for the majority of the size) is about 392MB. Most of the actual game code is in Managed/Assembly-CSharp.dll. This is 1.2MB, and a quick zip gets it to 470KB, pretty normal. There are a bunch of engine code dlls; the whole thing is about 13MB. The bulk of the game data is in StreamingAssets, about 375MB. This has files like characters, enemies, scenery, sound, etc. This means it has all the textures, models, sound, and stuff. A quick zip gets that to 230MB (7z with max compression down to 182MB), but as I explained earlier, there's always a tradeoff between performance and file size. In any case it's basically in-line with other engines.

Now, if we're done with Game Development 101 for today, an interesting post about where this might really be coming from (though I don't believe for a second that John Riccitiello is blameless). Short version, merging with sketchy ad companies and exposing yourself to investment vultures by going public might not be a great idea.

avatar
Dark_art_: 6 - Now that Unity is taxing per installation, the devs themselves, might want to know how many installs have been done. Games that had no reason for telemetry and data collecting will start to implement it and send to several parties.
Except that won't fly, for example Apple is very strict about data collection and even developers of games on the app store don't get anything except very high-level (i.e. vague) data about installs. They certainly aren't going to let Unity have any specific install data.

avatar
idbeholdME: Wait.... WHAT??? Reaching 200,000 instantly charges you for those 200,000 installs? It's not only the installs above that threshold? If so, then that's actually retarded.
No, that's not how it works. The whole thing is stupid and confusing enough as it is, but not quite that idiotic.

avatar
Lexor: I would like to receive some personal guidance "what now?" for the game project I'm working on but I'm not sure if I should ask about it here (as it can be easily buried under other subject-related posts) or maybe posting separate topic will be much better idea.
There are some signs that management might be willing to talk about the two major issues (retroactively changing TOS, using installs as a metric with zero transparency). I'd suggest waiting a few days to see if they have any kind of rational response, and if so finish your current project, but be prepared to move to a different engine afterwards. There are too many bad actors involved in running Unity now (see link above) who weren't there until recently, and even if they revert everything 100% this time, they'll just try something else when the uproar dies down.
avatar
Lexor: Well, I am not going to lie, this situation made me very very sad. :/

I would like to receive some personal guidance "what now?" for the game project I'm working on but I'm not sure if I should ask about it here (as it can be easily buried under other subject-related posts) or maybe posting separate topic will be much better idea.
I'd suggest looking at some basic tutorials / insight videos for other engines, seeing what might fit for you and coming back with a seperate thread.
avatar
EverNightX: I don't know of anything comprehensive. Off the top of my head I think Rayman Origins, Pathfinder, cuphead, Mimimi games (like Desperados III), and Solasta.
I don’t know what Rayman Origins engine is, but it is not Unity.
On the other hand, I can confirm that Pathfinder: Kingmaker and all stealth games by Mimimi (Shadow Tactics, Desperados III, Shadow Gambit) are built on top of Unity.
avatar
EverNightX: I don't know of anything comprehensive. Off the top of my head I think Rayman Origins, Pathfinder, cuphead, Mimimi games (like Desperados III), and Solasta.
avatar
vv221: I don’t know what Rayman Origins engine is, but it is not Unity.
On the other hand, I can confirm that Pathfinder: Kingmaker and all stealth games by Mimimi (Shadow Tactics, Desperados III, Shadow Gambit) are built on top of Unity.
Yes, I was wrong about Rayman.
avatar
Lexor: Well, I am not going to lie, this situation made me very very sad. :/

I would like to receive some personal guidance "what now?" for the game project I'm working on but I'm not sure if I should ask about it here (as it can be easily buried under other subject-related posts) or maybe posting separate topic will be much better idea.
Well, here's a bundle of all the Godot learning series if you're inclined to have someone else roll up your sleeves.
Disunity is more like it. Maybe they should change the name?
https://gamefromscratch.com/move-from-unity-to-godot-engine-in-seconds/

https://docs.godotengine.org/en/3.1/getting_started/editor/unity_to_godot.html