Jump to content

Photo

[Resource] Beyond 128: "Extended" Array Functions to go to 512 and up

Started by Chesko , Sep 29 2012 09:59 AM
Papyrus Array

  • Please log in to reply
16 replies to this topic

#1
Chesko
Posted 29 September 2012 - 09:59 AM

Chesko
  • Members
  • PipPipPip
  • Diviner
  • Joined: 13-July 10
  • 2634 posts
I have been getting annoyed with the Papyrus size limit of 128 elements on a given array, and on one of my mods, this has turned into a serious problem. So, I developed a set of functions identical in functionality to my general purpose array functions that allow you to treat multiple arrays like one large array, without having to worry about how to manage it yourself. You can just call ArrayAddFormXT, give it a form to dump in, and you can just trust that it will place the supplied form in the first element it can.

These example functions are a bit hard-coded to assume that you are using 4 individual arrays for a total of 512 elements. If you want more than that, it can easily be done, you'll just have to rewrite some parts. These might still be a little rough around the edges, so feedback is appreciated. However, they are all tested, and they all work (see test results below). All arrays in these examples store objects of type Form, but this can also be changed to your liking. The idea behind these functions and my general purpose array functions was to make arrays as easy to use as a FormList, which is what the functionality of these functions are tailored around.

Method:

The idea is that you are grouping together several arrays and referring to them in the future by an "Array ID", which is an integer. The first set of functions I defined are my "selector" functions that call the right "Do" functions based on the supplied Array ID. You will notice that the array names are baked into these function calls. I wish it could be made more generic, but since I wanted to keep the function calls short, these functions do have to have prior knowledge of the arrays you intend to use. In these examples, I use 8 arrays broken into 2 Array IDs, 1 and 2. They are: myArray1 through myArray8.

The below functions are what would actually be called by you when trying to interface with your Array IDs.

bool function ArrayAddFormXT(int ArrayID, Form myForm)
Adds a form to the first available element in the first available array associated with this ArrayID.
Spoiler


bool function ArrayRemoveFormXT(int ArrayID, Form myForm, bool bSort = true)
Removes a form from the arrays associated with the ArrayID, and re-sorting the arrays of the Array ID by default.

Spoiler


bool function ArrayHasFormXT(int ArrayID, Form myForm)
Attempts to find the given form in the given Array ID, and returns true if found.

Spoiler


function ArrayClearXT(int ArrayID)
Clears all arrays associated with this array ID.

Spoiler



int function ArrayCountXT(int ArrayID)
Counts the number of indicies associated with this array ID that do not have a "none" type.

Spoiler



int function ArrayCountFormXT(int ArrayID, Form myForm)
Counts the number of times myForm appears in arrays associated with this array ID.

Spoiler



bool function ArraySortXT(int ArrayID)
Removes blank elements by shifting all elements down, moving elements to arrays "below" the current one if necessary.
Spoiler


So, those functions wrap up your logic of calling several different Array IDs and knowing which arrays to pass into the Do functions when called.

Here are the "Do" functions. These perform the actual work by acting on the supplied arrays provided by the functions above. These would not be called directly in your code, unless you really wanted to (though that defeats part of the purpose of abstracting this down to a single Array ID, but it's your choice).


bool function ArrayAddFormXT_Do(Form[] fArray1, Form[] fArray2, Form[] fArray3, Form[] fArray4, Form myForm)
Adds a form to the first available element in the first available array.

Spoiler


bool function ArrayRemoveFormXT_Do(Form[] fArray1, Form[] fArray2, Form[] fArray3, Form[] fArray4, Form myForm, int ArrayID, bool bSort = false)
Removes a form from the arrays, if found.

Spoiler


bool function ArrayHasFormXT_Do(Form[] fArray1, Form[] fArray2, Form[] fArray3, Form[] fArray4, Form myForm)
Attempts to find the given form in the arrays, and returns true if found.

Spoiler


function ArrayClearXT_Do(Form[] fArray1, Form[] fArray2, Form[] fArray3, Form[] fArray4)
Deletes the contents of arrays.

Spoiler



int function ArrayCountXT_Do(Form[] fArray1, Form[] fArray2, Form[] fArray3, Form[] fArray4)
Counts the number of indicies in these arrays that do not have a "none" type.

Spoiler


int function ArrayCountFormXT_Do(Form[] fArray1, Form[] fArray2, Form[] fArray3, Form[] fArray4, Form myForm)
Attempts to count the number of times the given form appears in the arrays.

Spoiler


bool function ArraySortXT_Do(Form[] fArray1, Form[] fArray2, Form[] fArray3, Form[] fArray4, int i = 0)
Removes blank elements by shifting all elements down, moving elements to arrays "below" the current one if necessary, optionally starting the sort at position i.

Spoiler


Edit: Two more useful functions for getting rid of forms from mods that may no longer be present. See this post.

bool function ArrayRemoveBlankFormsXT(int ArrayID)
Clears all arrays associated with this array ID of blank forms ([Form <None>]) and re-sorts the array ID.

Spoiler



bool function ArrayRemoveBlankFormsXT_Do(Form[] fArray1, Form[] fArray2, Form[] fArray3, Form[] fArray4, int ArrayID)
Clears all arrays of blank forms ([Form <None>]) and re-sorts.

Spoiler



So, that's it. With the above, you can drop your forms into a set of arrays and not worry about managing them yourself, just like it were one big array, with the ease of use of a FormList.

Examples:
ArrayAddFormXT(1, FoodCabbage)	 ;Adds the form FoodCabbage to the first available element in the arrays associated with ArrayID 1.
ArrayRemoveFormXT(2, Gold001)	 ;Removes the first form matching Gold001 in the arrays associated with ArrayID 1.
bool bHasPlayer = ArrayHasFormXT(1, PlayerRef)	 ;Does Array ID 1 have the form PlayerRef?

So here's the big question. Does it work? The answer, thankfully, is yes. I wrote a test script to exercise most aspects of the above function calls; each test requires the function in question to look ahead to at least the next array, if not all 4 arrays in the Array ID.

Here are the test results. These tests walk all of the arrays in an Array ID (512 elements) several times to verify functionality. It also includes some performance test results, which I was pleasantly surprised and happy with. Trying to do everything that this does using FormLists would take forever, and much of the functionality of FormLists is broken anyway (which is what led me to write my general purpose array functions in the first place). Note: all of the below test results are from the same Papyrus log, but had to be broken up due to length reasons. The entire test took around 4 seconds to run.

Test Array Setup
Test 1 - ArraySortXT Test
Test 2 - ArrayAddFormXT Test
Test 3 - ArrayRemoveFormXT Test
Test 4, 5, 6: ArrayHasFormXT Test, ArrayCountXT Test, ArrayCountFormXT Test
Test 7 - ArrayRemoveFormXT Test (with Sort required)
Test 8 - ArrayClearXT Test
Test 9 - Array Population Test

Highlights from the tests:
1. Everything works. Yay!
2. ArraySortXT sorted 184 elements strung across 4 arrays in 1.533998 seconds. It managed to compact everything into the first 2 arrays after sorting it.
3. ArrayAddFormXT added a new form to the 2nd array of the Array ID in 0.016998 seconds.
4. ArrayRemoveFormXT removed a form at array 2, index 56 in 0.034000 seconds.
5. ArrayHasFormXT found a Gold001 form in array 2 index 12 in 0.034000 seconds.
6. ArrayCountXT counted 184 populated array elements across 4 arrays in 0.015999 seconds.
7. ArrayCountFormXT counted all 4 instances of Gold001 in 0.016998 seconds.
8. ArrayRemoveFormXT removed a Gold001 element from the middle of Array 2 and resorted the array in 0.232998 seconds.
9. ArrayClearXT cleared all 184 array elements by setting them to "none" in 0.015999 seconds.
10. ArrayAddFormXT populated all 512 elements of the Array ID with a FoodCabbage form in 2.250000 seconds, and gracefully returned "false" when all arrays of the Array ID were full.

Enjoy, and let me know if you have any questions or suggestions.

Edited by Chesko, 29 September 2012 - 03:44 PM.

#2
Chesko
Posted 29 September 2012 - 10:23 AM

Chesko
  • Members
  • PipPipPip
  • Diviner
  • Joined: 13-July 10
  • 2634 posts
Here is the test mod that runs the performance tests above.

The Nexus page includes a download to a test mod that contains the script files used in the test cases cited at the bottom of the post. It is complete: you can install it via Nexus Mod Manager, and it will execute the test cases on game start-up. Please see your Papyrus log for test results. You can use the included source file to copy functions into your own scripts.

#3
Chesko
Posted 29 September 2012 - 12:14 PM

Chesko
  • Members
  • PipPipPip
  • Diviner
  • Joined: 13-July 10
  • 2634 posts
Another example: Here is an actual implementation of ArrayAddFormXT(). This is also an example of how you could change the Array ID's to strings instead of ints, so that they might be a little easier to call and remember.

Spoiler


Now all I have to do to add, say, SuperWarmGauntlets to the "full" protection set of arrays for hand equipment is:

ArrayAddFormXT("hands full", SuperWarmGauntlets)

...without having to care which actual array it goes in.

Edited by Chesko, 29 September 2012 - 12:20 PM.

#4
Ingenue
Posted 29 September 2012 - 12:36 PM

Ingenue
  • Members
  • Pip
  • Curate
  • Joined: 05-March 12
  • 816 posts
  • Location:Kernow
Thank you very much for these!

#5
Chesko
Posted 29 September 2012 - 03:14 PM

Chesko
  • Members
  • PipPipPip
  • Diviner
  • Joined: 13-July 10
  • 2634 posts
Fun fact: If you populate a FormList with forms from a mod, and then remove that mod, your FormList will look like this:

Spoiler


That FormList had 72 entries for 72 hoods, but now that the mod is missing, those forms all return None. GetSize still reports 72, it doesn't get any smaller.

When you add something to that FormList, here's what it looks like (in this example I called _DE_WinterIsComingHoods.AddForm(ArmorCuirass) ):

Spoiler


Notice that the new item is now at the end of the list instead of reclaiming the space left behind by forms that are no longer present. Basically, once you populate a FormList, it's always going to be that size, and will only continue to get bigger. And since RemoveAddedForm() and Revert() are broken, FormLists should only be used if you do not intend to change the contents of the FormList at runtime, ever. For everything else, try to use an array, you have a lot more control over your data that way.

Edit: I just verified that if you re-introduce the mod back into the load order, the forms "take back" their appropriate "none" spots that they left behind in the FormList. So that's something, I guess.

With an array, under the same circumstances, those forms would have appeared as "[Form <None>]". If you check for that string in an array, you can actually get rid of it and revert it back to None and reclaim the space in your array. Here are a couple of functions to do that.

bool function ArrayRemoveBlankFormsXT(int ArrayID)
Clears all arrays associated with this array ID of blank forms ([Form <None>]) and re-sorts the array ID.

Spoiler



bool function ArrayRemoveBlankFormsXT_Do(Form[] fArray1, Form[] fArray2, Form[] fArray3, Form[] fArray4, int ArrayID)
Clears all arrays of blank forms ([Form <None>]) and re-sorts.

Spoiler




Here's the same scenario using an array. Here we have 72 [Form <None>] Forms, and then legitimate forms that we want to keep.

Spoiler


After calling ArrayRemoveBlankFormsXT()...

Spoiler


Now those elements are truly set to None, and we can use that space again using our other functions.

Edited by Chesko, 29 September 2012 - 03:54 PM.

#6
Cipscis
Posted 29 September 2012 - 11:25 PM

Cipscis
  • Members
  • PipPipPipPip
  • Master
  • Joined: 05-July 07
  • 6184 posts
  • Location:Auckland, New Zealand
This looks pretty nice, I've just downloaded the test file and I'm having a look through it. One comment I thought I'd make is that, since your _Do functions don't rely on any external variables, properties, or non-global functions, it would be best to make them Global functions. The main advantage of this, of course, is that many different scripts will be able to make use of them without having to worry about threading issues caused by a single object being accessed by multiple different object simultaneously. Perfect for a library such as this.

Global functions seem to have slightly less overhead than non-global functions too, even when threading isn't a concern.

I was actually working on something similar to this just the other night, although it didn't even approach completion. After seeing this I might end up making use of what you've written instead of reinventing the wheel myself :P

P.S. It looks like the forum has wiped some of your formatting - not all of your scripts are indented in your post, although of course when I look at the original script I can tell that they should be.

Cipscis

#7
Chesko
Posted 30 September 2012 - 08:21 AM

Chesko
  • Members
  • PipPipPip
  • Diviner
  • Joined: 13-July 10
  • 2634 posts
Great idea Cipscis. I'll see about making the change.

re: formatting, yea, unfortunately it doesn't appear like I can do much about it without having to manually add spaces everywhere :/

I just wish I could decouple my "selector" functions from their dependency on local arrays, but since I can't have an array of arrays, the only way to do it would essentially just call the _Do function straight. But that makes actually calling it in the code unwieldy and long and it starts to look inelegant. Ah well.

#8
SmkViper
Posted 01 October 2012 - 09:34 AM

SmkViper
  • Bethesda Game Studios
  • Programmer
  • Joined: 04-April 06
  • 350 posts
Hrm - it appears you're not aware of the array find/rfind functionality added in 1.6: Wiki link.

It should help speed up your code even more as you won't have to iterate through your large array in Papyrus code, but will be able to take advantage of native code speed to skips chunks of the array you don't care about.

Of course, always measure and compare, but you already know that ;)

#9
Chesko
Posted 01 October 2012 - 12:36 PM

Chesko
  • Members
  • PipPipPip
  • Diviner
  • Joined: 13-July 10
  • 2634 posts
Hey Viper,

No, I am fully aware of Find and rFind. However, all of the wiki examples imply that only strings can be searched for using either of these methods. Since I'm looking for objects usually of type Form, I didn't think it would work. If this does work on Forms without recasting them to a string first, perhaps the documentation needs to be edited? There is no mention what the valid input types into this method are.

Thanks for commenting!

Edited by Chesko, 01 October 2012 - 12:39 PM.

#10
SmkViper
Posted 01 October 2012 - 12:44 PM

SmkViper
  • Bethesda Game Studios
  • Programmer
  • Joined: 04-April 06
  • 350 posts
Ok, yeah, the wiki could probably be more clear on that, (strings are easy to use for examples, cause they have visible results) but the syntax callouts on that link do say "element type" rather then "string", so they *do* actually work for any array type Papyrus supports.

In short, you can use find and rfind for forms :smile:

(Plus the compiler is pretty good about complaining if you do something that won't ever work. For example, it will complain if you try to search an array of forms using a string, but will be just fine if you try to search an array of forms with a form)

#11
Chesko
Posted 01 October 2012 - 12:47 PM

Chesko
  • Members
  • PipPipPip
  • Diviner
  • Joined: 13-July 10
  • 2634 posts

Ok, yeah, the wiki could probably be more clear on that, (strings are easy to use for examples, cause they have visible results) but the syntax callouts on that link do say "element type" rather then "string", so they *do* actually work for any array type Papyrus supports.

In short, you can use find and rfind for forms :smile:


Awesome, thank you. I'll make the change and do a stare-and-compare of the same performance test posted above and see what the net gain is. Thanks for the help.

#12
Cipscis
Posted 01 October 2012 - 03:32 PM

Cipscis
  • Members
  • PipPipPipPip
  • Master
  • Joined: 05-July 07
  • 6184 posts
  • Location:Auckland, New Zealand
Just out of curiosity, how would the game react if I were to create large arrays by editing the assembly? I currently have access to the compiler and assembler, but not to Skyrim, so while I see that the assembler doesn't throw any errors for creating large arrays (although I wouldn't necessarily expect it to even if they won't work) I'm not sure if the game would reject them or if it might cause other problems.

I'm basically thinking about doing something like this:
Spoiler
The above will be assembled into a fully compiled script just fine, since it's basically bypassed the compiler's validation.Of course, I expect the validation in the compiler is there for a reason. I'm basically just wondering what sort of problems, if any, I might expect from such a function. Would it work? Would the game reject it outright? Might it cause other issues?

I notice that the assembler doesn't allow me to assembly a function returning an array of a length determined at run-time, although that seems to be a different sort of error (wrong type as opposed to argument out of range - error: Argument 2 must be an unsigned integer literal.)

Cipscis

Edited by Cipscis, 01 October 2012 - 03:41 PM.

#13
Chesko
Posted 01 October 2012 - 05:18 PM

Chesko
  • Members
  • PipPipPip
  • Diviner
  • Joined: 13-July 10
  • 2634 posts
I say you try it and find out Cipscis :)

#14
PhiniusMaster
Posted 02 October 2012 - 07:57 AM

PhiniusMaster
  • Members
  • PipPip
  • Disciple
  • Joined: 03-March 07
  • 1796 posts
This is great. Thanks a bunch!

#15
KhalEdos
Posted 28 October 2012 - 05:22 PM

KhalEdos
  • Members
  • Acolyte
  • Joined: 04-March 12
  • 151 posts
HI, All:) do you guys know of any way to make an array of objects created in one script visible or usable in another, like a formlist is?

#16
DocClox
Posted 29 October 2012 - 04:56 AM

DocClox
  • Members
  • Pip
  • Curate
  • Joined: 02-December 11
  • 603 posts
  • Location:UK

This looks pretty nice, I've just downloaded the test file and I'm having a look through it. One comment I thought I'd make is that, since your _Do functions don't rely on any external variables, properties, or non-global functions, it would be best to make them Global functions. The main advantage of this, of course, is that many different scripts will be able to make use of them without having to worry about threading issues caused by a single object being accessed by multiple different object simultaneously.


Won't that lead to threading issues though? Call a global function and other threads get a chance to run, which could result in some very counter intuitive behavior. You'd also get cases where more than one script might want to use the functions concurrently which could have a performance impact if I understand the threading model correctly.

#17
Chesko
Posted 29 October 2012 - 11:42 AM

Chesko
  • Members
  • PipPipPip
  • Diviner
  • Joined: 13-July 10
  • 2634 posts

HI, All:) do you guys know of any way to make an array of objects created in one script visible or usable in another, like a formlist is?


Form[] property myFormArray auto

Just define your arrays as properties instead of local variables. You should be able to access them externally.
Back to The Creation Kit

1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users

  1. Bethesda Softworks Forums
  2. → The Elder Scrolls
  3. → V - Skyrim
  4. → The Creation Kit