30 replies to this topic

[Lojack]

We're back, with an update on the information provided in [IMPORTANT] BSAs and You!

First off, make sure you've read [IMPORTANT] Load Order and You, by wrinklyninja.

With the latest Beta patch for Skyrim officially getting sent out to all clients, there have been some changes to BSA Load Ordering that you might want to know about. Basically, the issues brought up in the previous thread have been fixed. So this post is only here for educational purposes, in case you're curious as to the order of resources loading.

To summarize, the order resources are loaded is:
 

Registered BSAs -> Plugin BSAs -> Loose Files

With Registered BSAs being loaded in the order listed in Skyrim.ini, and Plugin BSAs loading in the same order as their plugins, which are loaded in the order they are listed in plugins.txt.

Disclaimer: The game doesn't actually work exactly like this. This is how it logically works. As in, if you had to describe the decision process the game goes through, this is it. Realistically, working this way would be a waste of resources and processing time (why load up every file, when you already know a different one is going to "win"?). It's just much easier to understand if you think of it like this. A good description of how it would actually work is here.

Contents:

• Resource Load Order
  • Registered BSAs
  • Plugin BSAs
  • Loose Files
• What This Means For Mod Users
• Other Questions
  • Replacing Scripts
  • Directory Thrashing
  • What are all these terms you use?
  • What about Nitpick?
• Addendum: BSA Compression

1. Resource Load Order
Skyrim loads its resources from the Data folder. Makes sense. These resources can also be packed up into a nice neat little archive, called a BSA. These are Bethsoft's custom archive files, and internally they mirror the layout of the Data folder. They're nice, because they keep the Data folder uncluttered, and it makes it really easy to remove every file from one mod.

Ok, but what happens when the same resource exists in more than one place? For example textures/dungeons/imperial/battlemap01.dds, which is the texture for the map you see all the time in castles, with the little flags and stuff sticking out of them. This file exists in Skyrim - Textures.bsa. That's the original file. If you download a texture replacer, you'll get it as a loose file probably, and if you download HD Skyrimmap from Steam Workshop, you'll get that file in a BSA. So which file actually gets used? Well, which ever one is loaded last. The game loads resources in three stages, with the last files loaded being what you see in game:

Needs Investigation: With pre-1.4.26 Skyrim, only Registered BSAs and Loose Files were loaded when the game first loads. That means anything in a Plugin BSA won't take effect yet - but you'll only notice that if you're trying to replace a Main Menu asset, like the logo. Once you start a game, either by loading a saved game or starting a new one, then the Plugin BSAs are loaded. So in the example of Brumbek's Main Menu Logo replacer, if it's loaded via a Plugin BSA (like it is when installed via Steam Workshop), you won't see the spinning logo until you load a game, then exit back to the Main Menu.

So the investigation needed: Is this still the case? I'll get back to you on that once I get the time to test it.

Registered BSAs
First the game reads your Skyrim.ini file. There's a section that tells the game which BSAs to load. It looks like this for most people:

[Archive]
sResourceArchiveList=Skyrim - Misc.bsa, Skyrim - Shaders.bsa, Skyrim - Textures.bsa, Skyrim - Interface.bsa, Skyrim - Animations.bsa, Skyrim - Meshes.bsa, Skyrim - Sounds.bsa
sResourceArchiveList2=Skyrim - Voices.bsa, Skyrim - VoicesExtra.bsa

The game will load the BSAs in order as listed, BSAs in sResourceArchiveList first, and sResourceArchiveList2 second. So if you manually register a texture replacer BSA (like the HD Texture packs), but put it at the beginning of the list - they won't load! Because the Skyrim - Textures.bsa will override them of course. If you put it after, or at the end of the list, or in the second entry, then they will load.

Easy enough, but there is one drawback: There's a 255 character limit on each of these entries. This is because the game internally uses a 256 byte character buffer to receive the value when reading it. One byte is reserved for the NULL terminator (a byte with a value of 0, indicating the end of the data), so that leaves 255 characters for file names. If you want to get around this limitation, you'll need to use Nitpick.

Plugin BSAs
Next the game loads any BSAs associated with a plugin. When the game loads an active ESP, it also looks for a BSA of the same name (For Oblivion, there were naming conventions that allowed a single ESP to load multiple BSAs in this manner - this is not the case for Skyrim). So dummy.esp will make the game load dummy.bsa. This is true even if the ESP does absolutely nothing (for example, the official HD Texture packs do this, as well as Oblivion's Shivering Isles expansion). Cool! We found an easy way to make our replacement textures only load when we want! Just deactivate the plugin and they don't load, activate it and they do!

Ok, but what happens if two Plugin-loaded BSAs have the same file in it? Well, it's easy now: whichever one loads last wins (as usual). But now there's a reliable way to control which order they're loaded in: they're loaded in the same order as their associated plugins. This means if dummy.esp loads before another.esp, then dummy.bsa will also load before another.bsa, meaning of course that another.bsa will win, because it loaded last.

Sounds good, but I seem to remember there being other issues with using BSAs...

Yes, nothing major really, unless you're trying to mod your game with a lot of plugins. Due to the way the game indexes plugins and stores the data in your save game, you can only have 256 active mod files. That's because objects are referenced by what's called a FormID, which is a 32-bit (4-byte) unique number, usually written as an 8-digit hexadecimal number. The first two hex digits are used to identify which mod the data came from. Two hex digits can represent a maximum of 256 unique numbers (0-255). Now, Skyrim.esm and Update.esm will always fill up two of those. Another one that's not obvious is your saved game itself - this one is always assigned the ID of hexadecimal FF, or 255. That leaves room for 253 other plugins. And here we get to our point: dummy plugins that don't do anything at all, but exist solely to make the game load a BSA? Yep, those take up one of our plugin slots. This might not seem like a big deal, but once you start trying to play with 300+ mods, you'll be faced with the hard decision of deactivating some mods. Luckily, dummy plugins you can just unpack the BSA and use the resources as Loose Files, then you don't need the plugin anymore. Other plugins can be merged via Wrye Bash into the Bashed Patch, further cutting down your mod count. And hopefully an equivalent of TES4Gecko will come out eventually, which will be able to merge mods into a single plugin. This last one however has an impact on your saved games, so it's for the more advanced user only.

Needs Investigation:Another small strangeness associated with Plugin BSAs, at least with pre-1.4.26 - they aren't loaded until you load either a saved game, or start a new game. That means if you're still at the Main Menu for the first time, any replacers loaded this way haven't taken effect yet! Woah! Not a big deal, unless you're trying to replace the Main Menu Logo...Oh wait. That mod works fine when installed as Loose Files, or if the BSA is registered in your Skyrim.ini, but it only works as a Plugin BSA after loading a saved game. This behavior still need to be verified now that 1.4.27 Skyrim is out.

Loose Files
This one's simple. If the file exists in the Data folder, it will always win. It will always be used, even if the same file exists in a Registered BSA or Plugin BSA. This is how most texture replacers work, and this is what mod managers like Wrye Bash or NMM are made for. Managing the Loose Files so you don't have to track them yourself. It's easy to keep track of which file came from which installer when you only have 3 mods. But when you get up to 150 mods, each with their own resources, a lot of them overlapping, a mod manager just takes the headache out of it.

2. What This Means For Mod Users
Not much really. There is one minor disadvantage to all this: Steam Workshoop uses BSAs by default, even for texture replacers that don't need a plugin. That means you end up with an ESP that is only there to load a BSA, taking up one of your 253 mod slots doing it. If you start getting close to the 253 mod limit, you can cut back by either:

• (Recommended) Unpacking those BSAs to Loose Files. A useful tool for unpacking the BSA is DDSopt. Once the BSA is unpacked, just pack up those Loose Files along with any documentation into an archive, and install with your favorite mod manager. Now you can deactivate the plugin, saving you a mod slot.
• (Not Recommended) Register the BSAs. This method you'll use your Skyrim.ini to load the BSAs instead of using a plugin. First you'll need to deactivate or delete the plugin. Next, you'll need to add the name of the BSA to your Skyrim.ini, in the following section:
  

  [Archive]
  sResourceArchiveList=Skyrim - Misc.bsa, Skyrim - Shaders.bsa, Skyrim - Textures.bsa, Skyrim - Interface.bsa, Skyrim - Animations.bsa, Skyrim - Meshes.bsa, Skyrim - Sounds.bsa
  sResourceArchiveList2=Skyrim - Voices.bsa, Skyrim - VoicesExtra.bsa

  The reason I don't recommend this method is that it's harder to change the order. This method you need to edit your Skyrim.ini each time you want to change the order, whereas with Method #1, you just either change the Installer Order of the package in BAIN, or in NMM. Also, you'll run up on the 255 character limit pretty quick, which means that you'll need to make use of Nitpick, which in turn requires SKSE. There's nothing wrong with requiring SKSE, but you'll have to wait for a SKSE update after each game update.

Again, each of these tricks only works if the plugin in question is an empty, dummy plugin, solely there for the purpose of loading the BSA. You can check this using TESSnip or another similar utility, or right click on the mod in Wrye Bash and select 'Mark Mergeable...' to find out if the plugin is empty. Wrye Bash will tell you if the plugin is empty, but it won't say anything at all relating to its "emptyness" if it's not empty.

3. Other Questions
Replacing Scripts
Pre-1.4.26 for Skyrim, you couln't replace a Vanilla Script file with one in a Plugin BSA. This is no longer true, it has been fixed, and script replacers should work as both Loose Files and Plugin BSAs now.

Directory Thrashing
What is it? link. Basically, simply having too many ESP, ESM, and BSA files (even inactive) in the data directory would cause Oblivion to go crazy, bringing your game to a crawl as disk I/O went through the roof. It's why Wrye Bash has the Auto-Ghost feature.

Does it affect Skyrim? Sort of, but not in the old sense of the term. There's a different problem with Skyrim, such that if you have over 508 mod files in your data directory, the engine just plain refuses to load any of them, active or not. To fix this, use Nitpick. Wrye Bash's ghosting will also work here since, Wrye Bash will not allow more than 255 active plugins at once, and ghost the rest, meaning you will at most have 255 files in the data directory with .esm or .esp extensions.

There's also the added problem of plugin INI files (which pretty much no one uses - no fix for this), and part of the old thrashing problem, where the game tries to read inactive plugins. For the last issue, again Wrye Bash's Ghosting feature still works good.

What are all these terms you use?
I used a bunch of terminology here, most of it old, but there's a couple new terms I made up for this explanation.

• Plugin - any .esp or .esm file.
• ESM - Elder Scrolls Master. Usually saved as a .esm file, which has the same format as a .esp file, only one bit (as in, one bit of a byte) is different.
• ESP - Elder Scrolls Plugin. Most mods are ESPs, but some are ESMs. Their format is the same, but the game handles them slightly differently in some specific cases.
• ESS - Elder Scrolls Save. Your saved games. This is loaded after all the ESPs and ESMs are loaded, so that its changes (like what items are in which containers) can override what the plugins say.
• BSA - Bethesda Softworks Archive. Sort of like a .zip file, but specific to Bethsoft's games, and optimized (sort of) for their specific usage.
• Load Order - The order that your mods (ESPs and ESMs) are loaded. When conflicts occur, the last loaded conflict "wins".
• Install Order - The order that files are installed into your Data folder. When multiple packages install the same file, the last installed package "wins".
• BSA Load Order - The order that the game engine loads BSAs. When conflicts in data files occur, the last loaded BSA "wins".
• Plugin BSA - A BSA that the engine loads because an ESP/ESM with the same name is active.
• Registered BSA - A BSA that the engine loads because it's listed in Skyrim.ini.
• Loose File(s) - A resource file in the Data folder that's "loose", as in not packaged into a BSA.
• FormID - A unique ID assigned to each "record" of a plugin. Each object in the game is defined by a record, and each record has a FormID. A FormID is 32-bits, with the first 8 bits defining which ESM or ESP the record belongs to.
• SKSE - Skyrim Script Extender. A nice utility that everyone should be using. It aims to extend the functions available to scripts written for Skyrim. It's also useful as DLL loader, if one wants to inject some code into Skyrim's game engine.
• SD - Script Dragon. Another DLL loader, but with a different goal in mind than SKSE. Both can work along side each other.
• Vanilla - A term referring to Skyrim as distribute by Bethesda and Steam. This means no modifications have been done. Alludes to the "plainness" of the vanilla ice-cream flavor.

What about Nitpick?
I'm still maintaining it. shadeMe may also release updates as well, he has access to the Skyrim Nexus page, so either he or I can update it. The Nexus page also has links to the source code, but here's shadeMe's source and my source, links in case the Nexus is down at the moment.

4.Addendum: BSA Compression
This question comes up often enough to deserve an answer here: What's better, compressed BSAs or uncompressed BSAs?

To quote Ethatron:

So lets see a little calculation for two operations of reading data from disk, uncompressed and then compressed:

• Harddisk-throughput: 20MB/s
• Memory-throughput: 2000MB/s
• zlib-throughput: 80MB/s
• uncompressed size: 100MB
• compressed size: 60MB

This yields:

• Reading 100MB at 20MB/s from disk takes 5s.
• Reading 60MB at 20MB/s from disk takes 3s, decompressing 60MB at 80MB/s in-memory takes 0,75s, so a total of 3,75s.

In case the data is very small it's probable that seeking time will start dominating the times:

• Reading 1MB at 20MB/s from disk takes 0,05s.
• Reading 0,6MB at 20MB/s from disk takes 0.03s, decompressing 0,6MB at 80MB/s in-memory takes 0,0075s, so a total of 0,0375s.

Seeking may be as much as 0,002s. Nevertheless it doesn't change the fact that reading compressed data is faster as long as the decompressor is faster than the disk. It's a rather theoretic notion, but when using compressed solid archives even the seek-times go down as the disc-head has to reposition a smaller distance while bulk-reading from the same archive.

Also, in reality with a modern system the decompression speed of zlib reach realistic 250-500MB/s (as you can see here), which can be far beyond any SSD-speed.

So for the most part, compression is good for you. There are a few cases where this might be detrimental though: At very high compression rates, it seems some files are affected - notably sound files seems to take on a "fuzzy" sound to them in many cases. Textures and Meshes should be fine going to max compression, but be sure to test things if you're going to up the compression on your BSAs.

Edited by Lojack, 16 November 2013 - 01:36 AM.

[Aenara]

Hilarious! Soon as I saw the new buttons I thought of your prior posts demystifying the priority process. Wondered when you'd get a chance to look over the update and here's the answer. Thanks for the continued vigilance!

Personally I sighed with great relief to have Loose Files back at the top of the heap. Can get a bit hectic without a manager but I prefer that degree of control. I'm also going to keep the HiRes packs registered. They've been around long enough to become defacto and it's two less references in ESPs and ESSs.

[Psymon]

And the bad news???

Seriously though kudos to Bethesda for listing to modders.

Wonder if they would update Oblivion and the Fallouts to work this way?

Thanks lojack for having a clear mind that can work all this out.

[frihyland]

It seems like Bethesda is on the right track. Thanks so much for figuring out and consolidating all this info.

[papill6n]

Thanks to Bethesda to have listen modders.

[SoSway]

Directory Thrashing
What is it? link. Basically, simply having too many ESP, ESM, and BSA files (even inactive) in the data directory would cause Oblivion to go crazy, bringing your game to a crawl as disk I/O went through the roof. It's why Wrye Bash has the Auto-Ghost feature.

Does it affect Skyrim? I don't know, I didn't really feel like making 400+ plugins to test this out. Not to mention that there was never a reliable number anyone could pin down that triggered it. But, if it does affect Skryim, there's another good reason to not use BSAs at all. Loose Files didn't trigger it for Oblivion, which is why a lot of modders didn't use them (BSAs).

This isn't the directory thrashing I remember from the Oblivion days, the directory thrashing was the result of just too many files in the /Data, including files the game never loaded like read me's and screenshot, jpgs, etc. The 400 esp/esm/bsa was a different problem, and not true directory thrashing. In fact to reduce thrashing many people started wrapping up mods with large resources in uncompressed bsa archives to reduce directory thrashing and decrease load time.

Some time ago I had problems with random crashes. I have investigated them and found that Oblivion sometimes starts to randomly searches all its directories, and if this takes too long, it goes haywire. Ever seen the recommendation to not have several hundreds unused .esp files in your data folder? Likely a naive observation based on this behavior. Now, as seen with procmon, Oblivion does not only search \Data\, but ALL folders in its install and mygames-directory. After I have seen this, and removed all junk (which were several ten thousand files...) the crashes were gone. Now OBMM stores omods in a subfolder of Oblivion's install folder, therefore omods are affected by this search, too.

Edited by SoSway, 29 February 2012 - 11:06 PM.

[BytesMage]

Seems like it is the same thrashing. I always kept my Oblivion folders and game save folders as clean as possible. I do the same with Skyrim. I move my screenshots, readmes, unused .esp, etc to another drive and back it all up. I don't feel that Skyrim is any different than Oblvion this way. Well I assume it is no different until I know it is. Besides it is just good computer maintenance anyway. This is why Wrye bash, like Mash before it saved many a mod user. With NMM I put all the mod infos and mods on another HD too. Clean as possible Skyrim directory always. I never had any CTD with Skyrim until lots of replacers and now esp were installed. That was the only difference I noticed from Oblivion. In Oblivion I had CTDs from the get go (clean install) it got better after Bash, BOSS, TESEdit made my mods clean. Also with Skyrim knowing what I know from years of Oblivion I make a list of mods that are loose files so I know what I have installed and in what folders so if NMM borks a uninstall which happened to me last night I can manually remove stragglers without any issues. It is easy to add lots of mods, but much harder to remove if lots of mods replace the same files. Anal? YES!!!

[SoSway]

I use wrye bash. <3 BAIN archives for showing conflicts and install order.

[vbgreen]

this explained alot of things i didnt understand. thank you.

[Devrin]

Outstanding work to all those involved on this. I don't get this detailed of a "heads-up" (if any) on changes at work, seriously lol. Thanks much for your time, effort, and general willingness to dive into this stuff!!

[SoSway]

Just a question, but does the Nitpick loader need to be updated for 1.4.27 patch?

[shadeMe]

Just a question, but does the Nitpick loader need to be updated for 1.4.27 patch?

I shouldn't think so. Check the log file to make sure.

[Winterlove]

When the game loads an active ESP, it also looks for a BSA of the same name (for the most part, some additional variation allows one plugin to load multiple BSAs, but that's not important for this discussion).

Is there anywhere I can find out something about the "additional variation" mentioned above? I'm trying to package loose files into multiple BSAs, but it doesn't seem to be working.

My specific situation is this. I have a plugin (call it wibble.esp) and it changes the appearance of some vanilla NPCs. The FaceGen data for the changed NPCs' has been generated and is packed into wibble.bsa. (I know there are issues with respect to exporting facegen data, but leave that to one side a moment). That setup works - the NPCs appearance is what I'd expect, no dark faces.

Now, what I want to do is extend this so that I can have multiple BSA files - wibble - Core.bsa, wibble - AltFirst.bsa, wibble - AltSecond.bsa and so on. The idea is that the Core BSA contains invariant assets, while the Alternates supply different variants of assets (difference versions of a texture, for example), and there would only be one Alternate. So, to test this I renamed wibble.bsa to wibble - Core.bsa. My understanding suggests that as the renamed archive begins with the same name as the plugin, its contents should be loaded.

Except this doesn't work. The changed NPCs have dark faces, which suggests that the facegen data contained in the archive isn't being picked up. Changing the archive back to wibble.bsa and it works again.

Can anyone tell me what I'm missing? Thanks.

Edit: fix apostrophe crime

Edited by Winterlove, 29 March 2012 - 03:01 PM.

[alt3rn1ty]

.

Edited by alt3rn1ty, 23 April 2015 - 05:46 PM.

[deaths_soul]

A well needed bump.

[alt3rn1ty]

.

Edited by alt3rn1ty, 23 April 2015 - 05:46 PM.

[Arthmoor]

This deserves to be pinned so it won't get lost.

[alt3rn1ty]

.

Edited by alt3rn1ty, 23 April 2015 - 05:46 PM.

[GamerRick]

You mention DDSopt, and I found BSAopt by the same author. Should we use both?

Thanks for the info!

[zone22]

Program do you recommend to repack BSAs?

Does uncompressed/lowest compression for BSAs or have loose files have any performance advantage? I'm specifically thinking of shuddering during cell transitions when even when having sufficient VRAM.

[Alexander J. Velicky]

I think BSAopt is the recommended repack utility.

[alt3rn1ty]

.

Edited by alt3rn1ty, 23 April 2015 - 05:45 PM.

[nico2137]

Emma and the team working on the Vilja Companion mod have a huge amount of fuz files in their bsa and stick with the official archivers default compression ratio, because they have a strong suspicion compressing them higher will cause game problems.

I have heard there are other files which can also be problematic if compressed too highly which BSAOpt is capable of.

I personally use the official tool provided with the CK to pack my bsa archives, and I can confirm that turning the compression flag ON makes strings files unreadable for the game ("Lookup failed !" messages ingame). Same problem with fuz files associated to shouts (CTD).

[Windzor]

For my own learning about .esp and .bsa . I read the OP about dummy .esps being used to load the .bsa files. What i want to know is can you just place all your files into the .esp, or if not what kind of files can you place in the .esp?

 

Still trying to wrap my head around this.

 

Thanks

[Alexander J. Velicky]

For my own learning about .esp and .bsa . I read the OP about dummy .esps being used to load the .bsa files. What i want to know is can you just place all your files into the .esp, or if not what kind of files can you place in the .esp?

 

Still trying to wrap my head around this.

 

Thanks

Think of it like this, there are three main areas of the game: The Engine, The Info, and the Assets. The engine is the hardcoded executable, it executes the code and runs in realtime to make the game happen. The info is the ESP/ESM file, it tells the engine how to run and how to use the assets it's provided with. And the Assets are the external files that are used by the Info and the Engine. Things like textures, meshes, scripts, sound files, etc.

 

You can equate it to building a house. The Engine are the workers that do the building. The info would be the blueprint, and the Assets would be the materials.

 

Much like building a house, you can't actually put any assets in the info. I.e. you cannot put files into the esp. The esp contains things that tell the engine how to run. Where things go, what things are called, how much damage weapons do, etc. You cannot put a texture file into an esp. You can have the esp tell the engine to use the texture. Just like how you cannot put wood into a blueprint. That makes no sense.

 

Hopefully that doesn't make it more confusing... The esp is simply a container of information that cannot be represented as an asset (Like a texture or voice file), but is also not hardcoded into the engine. (Like physics, or AI calculation information)

Edited by Alexander J. Velicky, 14 May 2013 - 05:13 PM.

[Stromgarde]

Very useful information in this thread, thank you.  Had been away from Skyrim for a while and I had remembered the prior issues with unintuitive resource loading, and glad to see this eventually got fixed.

 

Quick question...in Oblivion I renamed many mod BSAs to variations of DLCShiveringIsles - something.bsa so I didn't have a bunch of plugins who sole purpose was to load BSAs, I just had one, the DLCShiveringIsles.esp, to do all that.

 

Is this still a valid approach, or has it been mostly replaced by just extracting BSAs and installing resources loose?  I'm also wondering about the high-res texture pack DLC as an example; as seems kind of wasteful to use three plugin slots just for loading BSAs.  What do people typically do with these?  Extract them, rename them, or just leave them as is because Steam might do something odd if one of the official DLC files is renamed?  Thanks.

[deaths_soul]

Nope, that workaround no longer exists. The BSA now needs to have the exact same name as the plugin.

[Stromgarde]

Nope, that workaround no longer exists. The BSA now needs to have the exact same name as the plugin.

 

OK, thank you.  Too bad, that was a nice workaround.

[J.O.D.]

I use the HiRes BSAs together with ESPs. As I see it, most of mods are using BSAs anyway - not just for textures and models, but also for scripts etc. and quite a lot of mods do have just an ESP and a BSA file, mostly because of the Steam Workshop. The texture replacers either use loose files or have both versions available and in the end, you usually do not "waste" an ESP slot for loading BSAs, as the ESPs actually contain something. Also, registering the BSA replacers in the INI file does work for some (rare) instances, where there is no ESP in the mod.

[Lojack]

Updated some stuff here:
- updated some links
- fixed some typo's/formatting
- adding info about directory thrashing

Edited by Lojack, 16 November 2013 - 01:39 AM.