Talk:Firing Accuracy Testing - Revision history

Bomb Bloke: /* Comments */

2009-03-15T14:20:44Z

Comments

← Older revision		Revision as of 14:20, 15 March 2009
Line 208:		Line 208:

	[[User:Spike\|Spike]] 09:32, 14 March 2009 (EDT)		[[User:Spike\|Spike]] 09:32, 14 March 2009 (EDT)

			----

			I, personally, am a great believer in the "angular accuracy" theory, and believe I've proved it on my [[User_talk:Bomb_Bloke#Firing_Accuracy\|talk page]].

			Furthermore, I do not believe the game ever decides whether a bullet will hit a target prior to the event actually happening. It determines the angle, sends the bullet on it's merry way, and whether it'll hit the target or not depends entirely on the "angular accuracy" formula and how big the target is.

			I was on the way to working said formula out but I kinda got sidetracked (as I so often do) and stopped generating the numbers needed to pull it off. As it stands, it'd probably take me a week of overnight logging to get the sort of figures I reckon are required.

			Though I admit part of that has to do with the hope that Seb will save me all the trouble and simply locate (& translate!) the code in the executable. I'm more about "working out the best way to do things" then actually ''doing'' them, and in this case, I honestly believe that'd be the most direct and accurate route. Call me lazy if you will. ;)

			- [[User:Bomb Bloke\|Bomb Bloke]] 10:20, 15 March 2009 (EDT)

Spike: /* Comments */ Possible "single accuracy function" and explanation of why Average Hit% fell below FFA

2009-03-14T13:32:00Z

Comments: Possible "single accuracy function" and explanation of why Average Hit% fell below FFA

← Older revision		Revision as of 13:32, 14 March 2009
Line 191:		Line 191:

	Thanks for coming back to me Mike! [[User:Spike\|Spike]] 08:43, 14 March 2009 (EDT)		Thanks for coming back to me Mike! [[User:Spike\|Spike]] 08:43, 14 March 2009 (EDT)

			:Something's just occurred to me that might explain your unexpected result that the Average % Hit did not fall to FFA%, but fell below FFA to about 5/6ths FFA. (At FFA of 45%, average hits looked like they fell off to 39% "at infinity"). Maybe FFA is the chance to hit the '''square''' at some notional range? Maybe your target occupied about 39/45ths of the cross sectional area of the square (especially if we ignore the vertical component, which seems to be less important).

			:In which case, maybe there is a "single accuracy function" that does something like this (for the horizonal plane):

			My FFA is "X".
			Therefore, I construct a firing cone of angle "A",
			such that, at a notional distance "D",
			X% of the firing cone angle = the width of one square.
			(X% A = inv tan (0.5/D)).
			I then generate a bullet on a random angle within that firing cone.

			:Actually 39/45 is exactly 5/6 - which is a suspicious number. The maximum cross section of an object in a tile is 16x24 = 384 pixels. Maybe your target had a 320 pixel cross section (5/6 x 384)? Or maybe there's a much simpler explanation, some kind of 5/6 "dice roll".

			:It would be very interesting to repeat the tests against targets of different size. At the very least, you would expect the "misses that hit" to be inversely proportional to some dimension of the target - width, or cross sectional area. Or in fact, inversely proportional to the "angular" width/area that they present to the firer at that distance.

			[[User:Spike\|Spike]] 09:32, 14 March 2009 (EDT)

Spike at 12:43, 14 March 2009

2009-03-14T12:43:57Z

← Older revision		Revision as of 12:43, 14 March 2009
Line 173:		Line 173:

	Good to talk with you again! -[[User:MikeTheRed\|MikeTheRed]] 21:13, 13 March 2009 (EDT)		Good to talk with you again! -[[User:MikeTheRed\|MikeTheRed]] 21:13, 13 March 2009 (EDT)

			Well I must admit I'm slightly daunted by that! I downloaded BB's fire testing tool but I haven't set it up yet - it does seem extremely cool. If you could upload your .mdb file I will take a look and see if I feel up to the task. :)

			From the SC thread, it looks like a conclusion was reached that there is not a hard distinction between 'hits' and 'misses', with a hit occurring below FFA% and a 'miss' being generated above FFA% - what you called a 'deliberate miss'. That instead, you just have a single (hidden) accuracy function that generates a vector for the bullet, and that vector sometimes intersects the target - roughly (9/10)xFFA% of the time at extreme range. This sounds like a cleaner system. The one pixel LOFTEMPS test would be a good critical test of that. Actually it might have to be a 2x2 target, since with a 1 pixel target a line through the integer-rounded, computed centroid of the target might pass exactly through one of the corner vertices of the 1 pixel target - similar to the diagonal wall bug. That's testable though.

			You could even test the size (and shape) of the cone angles by creating a series of concentric targets, of increasing diameter, one behind the other. You're talking zillions of test runs though, unless you just use that to verify an existing model that seems pretty solid already.

			Because I can't quite believe the levels of staggering genius that would be required to concoct and tune a "single function" for hits and misses, I'm still slightly leaning toward a 'deliberate hit' vs 'deliberate miss' model, where if a shot 'rolls' below FFA% (or some lower value), it's a 'perfect' shot, and if it rolls above that %, some error is introduced (either angular error, or perpendicular error per distance). Of course, we have been surprised before about the staggering level of detail under the hood in this game (e.g. LOFTEMPS), so it's perhaps not best to underestimate the designers.

			One good result in the SC thread was that you guys did find that the error cones were smaller for good marksmen. It would be interesting to investigate if this greater error was simply due to the greater range of "amount missed by" that is experienced whenever FFA is low. For example if my FFA is only 20%, I can "miss by" up to 80, whereas if FFA is 90, I can "miss by" only up to 10. But alternatively, the error angle could be a direct function of FFA (or WA, or FA): I missed, and my FFA is 20, so my error cone is (for example) 120 degrees minus my FFA. Anyway just speculation.

			You are definitely good on the statistics. I misunderstood why you were testing near to 100% but re-reading your notes in more detail, I can see what a cunning statistical exercise it was. For analysis like that I would have to let Excel do the work for me, as I don't have the knowledge. (Or maybe I'll download a trial version of SPSS :) )

			One point that occurred to me on the test setup methodology. You were pushing the damage up to 255 on the Laser Rifle to avoid zero damage rolls. But instead you could use an HE weapon (of power=2 perhaps) to guarantee 1-3 damage without affecting any other results. Especially since your target is suspended in mid-air.

			Well I ''might'' just give this ago, starting with the "zero FFA" test case, and maybe the "2x2 pixel target" test case.

			Thanks for coming back to me Mike! [[User:Spike\|Spike]] 08:43, 14 March 2009 (EDT)

MikeTheRed: /* Comments */ Reply to Spike

2009-03-14T01:13:24Z

Comments: Reply to Spike

← Older revision		Revision as of 01:13, 14 March 2009
Line 147:		Line 147:

	: I read the Strategy Core thread you linked to on the main article, and it addresses most of these suggestions and comments - and many more. [[User:Spike\|Spike]] 17:11, 13 March 2009 (EDT)		: I read the Strategy Core thread you linked to on the main article, and it addresses most of these suggestions and comments - and many more. [[User:Spike\|Spike]] 17:11, 13 March 2009 (EDT)

			-----

			As shown in that thread, BombBloke made an insanely cool tool for automated data collection; you let your PC run overnight while it collects data on hits and misses, noting the exact tile hit. Unfortunately as I said in my Jan 9 2007 post in that thread, I collected tons of data (36k points) which I was having trouble finding time to analyze - and then discovered I had been using an older map (without a ceiling) and for best results, should start over. Finally I found a major discrepancy between my first big set of automated data and second big set, when they should have found the same thing - something was very wrong (or misunderstood) across sessions, which threw the quality of all the data into doubt. Was I gonna have to do lots of big sessions just to figure out which of my sessions had good data and which had bad, nevermind actually getting any results?

			Between these problems, I simply stopped. After you put all that time into the setup - sorry, BB!

			Note that even once you have a ton of data, there is still a lot of work to be done, such as the fact that the raw data shows which tiles were hit (wall behind target, ceiling, and floor), but ultimately you want to transform that "awkward 3D data" (different tile orientations, widths, etc.) into a nice theoretical flat bulls-eye dartboard type distribution behind the target. It's hardly impossible, but it ''is'' work.

			I got his tool after writing the [[Firing Accuracy Testing]] page (that's why the link to BB's tool is the last thing on the page, then I eventually dropped it as I just said), and it's certain that the results of the automated testing (which included actual tile-hit and angle data) would have made my previous work look like scribbles on a napkin. If you're interested, please go for it! Myself and probably BB will be happy to help you get the setup and Autohotkey working. Hell, AutoHotkey is really cool for almost any game - you can e.g. switch any hotkey to be anything else, even if a game doesn't allow it. I've used it just to switch key assignments on lots of games.

			In retrospect I think it would be cool to try variations on the target, e.g., what if it's the smallest possible LOFTEMPS (1x1 pixel silhouette); do you still get a fixed amount of "guaranteed" hits, like I postulated? Versus other simple geometric targets like 10x10 pixels (or whatever), or the entire silhouette (what's that, 12x16 if memory serves? ... this would actually be a test that straight lines are followed and the tile directly behind the alien does not take a single hit), or the entire silhouette with a small opening, etc. etc.

			Your idea of zero accuracy is also a great one. In fact there's a lot of things that could be tried. But I've become too busy with other games to sink much time into XCOM these days. Just ask Zombie... he's been waiting months for me to put in a few more hours on finishing some WORLD.DAT work (sorry, dude! I will get to it). I still don't mind talking a lot though, lol.

			One other thing about this Firing Accuracy work is that I'm not good with trigonometry. I'm good with data, modelling, and probability, but not so good with angles (and 3D stuff, when relevant). Versus e.g. Zaimoni (a wiz with trig), Pi Masta (with his [[WORLD.DAT#See_Also\|WORLD.DAT world map]]), BombBloke (as seen in that thread and e.g. his [[User_talk:Bomb_Bloke#Firing_Accuracy\|Talk]] page), NKF, and others. While this is ok - me, them, Zombie, Danial, we all bring skills (from the mundane to the ultra cool) that work together to make this place an amazing repository of XCOM info - it still made this particular work a little more of a challenge to me.

			I see what you're saying, angular versus perpendicular. In my testing, I was actually watching the screen while I rapid-fired shots, as shown in the [[Media:FA_TestingSetup1.png\|screenshot]]. Using the DOS version in [[Info#Working_In_Windows\|DosBox]], I could increase its running speed and shoot as fast as I could click (and check UNITREF.DAT later for the number of hits). I don't think it can be cranked up in the Win CE edition (but could be wrong). Anyway, in this setup, there was no way to keep track of the distribution of misses, except to make a mental note of how wide the "farthest" misses went. (Conversely, BB's automated sampling setup is a natural for capturing the actual tiles hit.)

			Still, one thing is clear: whether it's angular or perpendicular, the equation also has a term for ''distance''. Otherwise, Test 1 would have shown an ultra-simple (straight line) relationship versus spread, when I changed the distance between shooter and target.

			If you want my old data, I can send it to you (4 MB .mdb, zips to 537k).

			Note: You're right, normally I wouldn't want to test near 100% except I was doing something different in Test 3 - I wanted as small variability in the sampled data as possible so that any differences would be the most evident. It wasn't regular firing accuracy testing; it was seeing if the WA and SA contributions to FFA mattered. Still, it could be done at 50% FFA too, shrug.

			Good to talk with you again! -[[User:MikeTheRed\|MikeTheRed]] 21:13, 13 March 2009 (EDT)

Spike: /* Comments */ note about SC thread

2009-03-13T21:11:25Z

Comments: note about SC thread

← Older revision		Revision as of 21:11, 13 March 2009
Line 145:		Line 145:
	Anyway, awesome work MTR.		Anyway, awesome work MTR.
	[[User:Spike\|Spike]] 16:18, 13 March 2009 (EDT)		[[User:Spike\|Spike]] 16:18, 13 March 2009 (EDT)

			: I read the Strategy Core thread you linked to on the main article, and it addresses most of these suggestions and comments - and many more. [[User:Spike\|Spike]] 17:11, 13 March 2009 (EDT)

Spike: New section: Comments

2009-03-13T20:18:50Z

New section: Comments

← Older revision		Revision as of 20:18, 13 March 2009
Line 114:		Line 114:

	There is something weird about this talk page though. I can't edit the section above, and when I edit the whole page, the section above does not exist! [[User:Spike\|Spike]] 16:15, 13 March 2009 (EDT)		There is something weird about this talk page though. I can't edit the section above, and when I edit the whole page, the section above does not exist! [[User:Spike\|Spike]] 16:15, 13 March 2009 (EDT)

			== Comments ==

			Hi MTR - this is great work, I only just stumbled across it. As I noted in linking it to [[Weapon Analysis]], it has implications for my firepower model. In my model I take 2 extreme cases: "Shock" where accuracy doesn't matter at all, and "Skirmish" where accuracy is critical. I then lamely say that real situations lie between the two. You have mapped out that ground here (and 2 years ago!).

			Since when you did this research, the LOFTEMPS understanding has improved and would probably allow you to determine the actual width of a Muton, plus any "holes" (crotch or otherwise). If I was re-running this test, I would try a test at '''zero''' FFA. As per your comments, it's a way of putting a floor in, it would give the pure random element isolated from any skill element. And incidentally it would also provide a useful (?) model of reaction fire by [[Talk:Alien_Inventory_Use#Melee_Aliens_With_Guns\|gun toting melee aliens]].

			I would also shy away from testing when FFA is up near to 100%. Since by definition you can't observe greater than 100% success, you are potentially 'clipping' the top side of the (observed) function.

			Excellent idea to reverse WA and FA and prove you get the same results, by the way - very rigorous.

			If I had to guess how the model worked, I would guess the engine introduces a random '''angular''' error to any miss (anything failing FFA roll).

			For example, your observed (and modelled) max spread of 5 at range 10, if that means max of 2.5 left or 2.5 right, another way of putting that is up to 14 degrees left or 14 degrees right, a total 28 degree cone (in the horizontal plane). Now, in the worst case, did the game engine decide to go 2.5 squares left per ten squares forward? Or did it decide to go 14 degrees left?

			From what I understand of your model, you're assuming a "perpendicular" error, where there is a "left or right by up to X" error and X is a direct proportion of the distance. It should be feasible to tell if the error is angular or perpendicular. If it's an angular error, e.g. "I missed so I will now roll a random +/- 0-20 degrees left or right", then the distribution of the error left or right on the X axis will approximate a trigonometric '''tangent''' function. If the error mode is perpendicular, "I missed so I will now roll a random +/- 0 to (distance x X) left or right", then that will produce a linear distribution of error left and right.

			Once the "dice are rolled" these amount to the same thing, and in the game engine once bullets are in flight they are probably implemented as X,Y,Z coordinate system vectors rather than as angles.

			If you visualised this on the "wall behind the Muton", with perpendicular error the bullet holes are uniformly distributed, with angular error they are clustered around the Muton and spread further apart, the further they are away from the Muton (target).


			Also if you could study the distribution of this error you could determine if it was genuinely random, or was related to (1-FFA), i.e. the "lack of accuracy" of the firer/weapon combination. In other words, do bad marksmen have wider error cones than good marksmen?

			Surprisingly, from your work it also looks like the engine sometimes or always introduces a smallish error into accurate shots - enough to take 1/10th of the expected accuracy off at longer ranges. This could be a deliberate simulation of 'windage', or it could be truncation/precision error - both as suggested by NKF.


			Also, as we all know from experience, there is a vertical error component as well. This might well explain some of the deviations between model and experiment. But its worth seeing if angular vs perpendicular randomness is the explanation. If you still have your raw test data we could take a look at it. Actually the worst imprecision in the data probably comes from the fact that an observed 'hit' is 'rounded' to the nearest 'integer' map square. I bet this bleeds off a lot of precision that actually exists within the game engine, and would be observable by us if we only had a way to log the pixel-path traced by the bullet, rather than just its crude endpoint. If we were going completely mad, we could use LOFTEMPS to build a sort of diffraction grating, slicing squares up into tenths or hundredths with part impenetrable wall, part air gap, and some kind of Big Red Death Tile in the next square over so you could clearly see when it passed through the air gap. But that's getting a bit mad!

			Anyway, awesome work MTR.
			[[User:Spike\|Spike]] 16:18, 13 March 2009 (EDT)

Spike: New section: Link is working - shame about the page

2009-03-13T20:15:47Z

New section: Link is working - shame about the page

← Older revision		Revision as of 20:15, 13 March 2009
Line 108:		Line 108:

	- [[User:Zaimoni\|Zaimoni]] 7:54, 28 November 2006 (CST)		- [[User:Zaimoni\|Zaimoni]] 7:54, 28 November 2006 (CST)

			== Link is working - shame about the page ==

			I wanted to update to say the link to the Strategy Core thread is fine, my network card was playing up.

			There is something weird about this talk page though. I can't edit the section above, and when I edit the whole page, the section above does not exist! [[User:Spike\|Spike]] 16:15, 13 March 2009 (EDT)

Zaimoni at 13:46, 28 November 2006

2006-11-28T13:46:25Z

← Older revision		Revision as of 13:46, 28 November 2006
Line 104:		Line 104:

	- [[User:MikeTheRed\|MikeTheRed]] 23:36, 27 November 2006 (PST)		- [[User:MikeTheRed\|MikeTheRed]] 23:36, 27 November 2006 (PST)

			Yes, that should be what's happening: at range one, a lot of angles map to the same pixel after integer math.

			- [[User:Zaimoni\|Zaimoni]] 7:54, 28 November 2006 (CST)

MikeTheRed at 07:36, 28 November 2006

2006-11-28T07:36:30Z

← Older revision		Revision as of 07:36, 28 November 2006
Line 96:		Line 96:

	- [[User:Zaimoni\|Zaimoni]] 22:02, 27 November 2006 (CST)		- [[User:Zaimoni\|Zaimoni]] 22:02, 27 November 2006 (CST)

			----

			I found it very odd that FFA 45 to 65 at Range 1, all had extremely few misses. Considerably fewer than the percent of misses that occurred, at any farther range. They did not seem to follow a decreasing pattern. I'm wondering if there was "exactly one pixel" (or whatever) somewhere odd, like right beneath the crotch or whatever, that was allowing a bullet through, ever so occasionally. And when I went to FFA 66, the cone of fire for "misses" finally was small enough to not include that one little "window" through the Muton. This could explain why it was so small, and also so invariably small across a range of FFAs.

			Here's a [[Media:FA_TestingSetup1.png\|screenshot]] showing my testing setup. For farther distances, my guys always move straight back. The facing and orientation for BB's fire-log map is the same. There might be testing of diagonals and other facings in the future, but for now I'm keeping it simple.

			- [[User:MikeTheRed\|MikeTheRed]] 23:36, 27 November 2006 (PST)

Zaimoni at 04:01, 28 November 2006

2006-11-28T04:01:06Z

← Older revision		Revision as of 04:01, 28 November 2006
Line 88:		Line 88:

	- [[User:Bomb Bloke\|Bomb Bloke]] 19:30, 16 November 2006 (PST)		- [[User:Bomb Bloke\|Bomb Bloke]] 19:30, 16 November 2006 (PST)

			----

			Just saw the updated thresholds. What's going on at range 1 is that the lowest FFA whose frustrum is completely intercepted by the Muton "in standard facing" is FFA 66...that is, all of the "accidental hits" are actually happening. As such, intensive testing is in order (this is the test case that gives us useful geometry information on the frustrum).

			Incidentally: perhaps you should document the standard facing? [I would think that what would matter is "relative facing" and whether the distance 1 was horizontal/vertical or diagonal. 16 test cases at range 1....]

			- [[User:Zaimoni\|Zaimoni]] 22:02, 27 November 2006 (CST)